mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-12-22 16:17:55 +08:00
Code Issues Packages Projects Releases Wiki Activity
5,700 Commits 32 Branches 296 Tags 152 MiB
9524b08bec
Commit Graph

3655 Commits

Author SHA1 Message Date
Chris Lu
217e0f9066 mount: remove folder recursively 2021-04-15 01:51:10 -07:00
Chris Lu
3e669e6d7b mostly refactoring, add some error handling 2021-04-14 23:33:37 -07:00
Chris Lu
07f712c83f fix typo 2021-04-14 23:21:38 -07:00
Chris Lu
36c79de3f4 fuse mount: dir ReadDirAll avoid extra conversion to filer_pb.Entry 2021-04-14 23:21:24 -07:00
Chris Lu
e41766feb6 fuse mount: dir lookup avoids extra conversion to filer_pb.Entry object 2021-04-14 22:38:34 -07:00
Chris Lu
1adc8f86ea lighten up File object 
file.entry only exists when file.isOpen
 2021-04-14 20:49:15 -07:00
Chris Lu
6bc09b18c4 truncate is a bit faster to reuse the storage 2021-04-14 20:26:56 -07:00
Chris Lu
c04b7e106f mount: remove entry from Dir object 2021-04-14 20:26:13 -07:00
Chris Lu
fc0cbf565f add option to obfuscate the file names 2021-04-14 15:37:24 -07:00
Chris Lu
e75633c64f volume.check.disk: break loop for read only volumes 
fix https://github.com/chrislusf/seaweedfs/issues/2002
 2021-04-14 12:40:13 -07:00
Chris Lu
9d50867d08 volume.tier.move: avoid data loss when destination volume server already has the volume 
fix https://github.com/chrislusf/seaweedfs/issues/2001
 2021-04-14 10:26:26 -07:00
Chris Lu
ff4c1d5965 adjust logging 
fix https://github.com/chrislusf/seaweedfs/issues/1999
 2021-04-14 10:04:26 -07:00
qieqieplus
270645f8d7 fix #1996 2021-04-14 18:29:28 +08:00
Chris Lu
90677e1097 ensure to call line.Close() 
fix https://github.com/chrislusf/seaweedfs/issues/1995

similar to https://github.com/peterh/liner/issues/104
 2021-04-14 01:33:21 -07:00
Chris Lu
3f3268cd1b go fmt 2021-04-14 00:30:16 -07:00
Chris Lu
ca0f07a188 move file reader, entryViewCache to file handle 
reduce file object size
 2021-04-14 00:29:58 -07:00
Chris Lu
5985a7d38d add log level possible values 
fix https://github.com/chrislusf/seaweedfs/issues/1989
 2021-04-12 21:15:51 -07:00
Chris Lu
f5de42fae3
Merge pull request #1975 from kmlebedev/iam_handlers 
IAM handlers
 2021-04-12 12:07:45 -07:00
Chris Lu
1e033d45b8 simpler logic 
related to https://github.com/chrislusf/seaweedfs/pull/1981
 2021-04-12 12:04:53 -07:00
Chris Lu
52200a903b
Merge pull request #1981 from ueni-ltd/if-modified-since 
Fix If-Modified-Since behavior
 2021-04-12 12:02:54 -07:00
Chris Lu
519b0e1e49 filer: upload to a directory without "/" suffix 
fix https://github.com/chrislusf/seaweedfs/issues/1988
 2021-04-12 11:56:56 -07:00
Konstantin Lebedev
8e02e138ea Merge branch 'upstreamMaster' into iam_handlers 2021-04-12 11:22:51 +05:00
Chris Lu
742ab1ec81 2.39 2021-04-11 19:47:11 -07:00
Chris Lu
0df5b53ad8 adjust help message 2021-04-11 00:26:28 -07:00
Chris Lu
f62c153274 go fmt 2021-04-10 23:48:18 -07:00
Chris Lu
af313dff58 add gateway for easier POST and DELETE blobs 2021-04-10 23:47:47 -07:00
Konstantin Lebedev
5021bea698 GetUserPolicy 2021-04-10 23:57:45 +05:00
Chris Lu
0b82edc0d2 filer: avoid stuck uploader 
fix https://github.com/chrislusf/seaweedfs/issues/1980

reverting the file upload batch executor
 2021-04-09 13:05:15 -07:00
Chris Lu
93f4146ffa properly release the view cache 2021-04-09 12:36:39 -07:00
Merlin Gaillard
f952f979d1 filer: return 304 when If-Modified-Since == Last-Modified 2021-04-09 15:04:17 +02:00
Merlin Gaillard
4d4acc715e s3api: handle 304 response code from filer 2021-04-09 12:13:19 +02:00
Chris Lu
6deb647a8f mount: fix possible memory leak 
if many files are read repeatedly, their metadata are accumulated in memory. This fix cleared the metadata after the file is read.
 2021-04-08 19:47:31 -07:00
Konstantin Lebedev
ba175f81b5 add auth aws signV4 2021-04-08 17:40:47 +05:00
Konstantin Lebedev
995ae91007 add DeleteUserPolicy 2021-04-08 11:16:36 +05:00
Philippe Pepiot
42a761ee20
Fix typo in weed filer long help 2021-04-07 22:47:23 +02:00
Chris Lu
3be061994f skip connection reset error 
fix https://github.com/chrislusf/seaweedfs/issues/1971

this is because the connections are pooled but the volume server has reset the connection
 2021-04-07 00:54:13 -07:00
Chris Lu
c5b08bac1b remove mac specific mount options 2021-04-07 00:54:13 -07:00
Konstantin Lebedev
f5f8eec8e2 fix get filerGrpcAddress 2021-04-06 13:53:56 +05:00
Konstantin Lebedev
011e6e90ee Merge branch 'upstreamMaster' into iamapipr 2021-04-06 13:50:33 +05:00
Konstantin Lebedev
ed79baa30f add tests 2021-04-06 13:43:08 +05:00
Chris Lu
2327c0756b fix to avoid loop 2021-04-05 23:24:26 -07:00
Chris Lu
a37eca78cd 2.38 2021-04-05 19:41:54 -07:00
Chris Lu
c131764c34 ensure error is sent back in channel 
fix https://github.com/chrislusf/seaweedfs/issues/1966

avoid shared readErr variable
 2021-04-05 19:40:12 -07:00
Chris Lu
0f64f5b9c8 mount: add readOnly option 
fix https://github.com/chrislusf/seaweedfs/issues/1961
 2021-04-04 21:40:58 -07:00
Chris Lu
6eee200c13 2.37 2021-04-04 18:45:48 -07:00
Chris Lu
fbb82a5c9c skip limiting if limit is zero 2021-04-04 18:38:33 -07:00
Chris Lu
8251d1140e refactor 2021-04-04 18:38:33 -07:00
Chris Lu
5dca98fe71 fix typo 2021-04-03 11:37:13 -07:00
Chris Lu
5e64f65632 ensure tables are created 
fix https://github.com/chrislusf/seaweedfs/issues/1957
 2021-04-02 12:02:26 -07:00
Chris Lu
319743d330 go fmt 2021-04-02 02:22:59 -07:00
Chris Lu
8e404a1433 go fmt 2021-04-02 02:22:26 -07:00
Chris Lu
bdf2ddddfd revert to same implementation as before 
This reverts commit 7e8edc3c4a.
 2021-04-02 02:21:38 -07:00
Chris Lu
67e019d54b add missing changes 2021-04-02 01:11:44 -07:00
Chris Lu
7e8edc3c4a refactoring 2021-04-02 01:10:24 -07:00
Chris Lu
cc0df36a9e smaller file chunks are faster 2021-04-01 02:21:59 -07:00
Chris Lu
cefe66f159 dedicated upload processor 
avoid thundering effect of overloading volume servers
 2021-04-01 02:21:40 -07:00
Chris Lu
1f984d2645 refactor buffer pool 2021-04-01 02:20:00 -07:00
Chris Lu
b5880334fc refactor 2021-03-30 21:07:34 -07:00
Chris Lu
e79e2ddeed remove unused MoveEvents 2021-03-30 20:42:44 -07:00
Chris Lu
e06807a55b adjust logs 2021-03-30 20:36:06 -07:00
Chris Lu
ac875976c0 filer, volume: add concurrent upload size limit to avoid OOM 
add some back pressure when writes are slow
 2021-03-30 02:10:53 -07:00
Chris Lu
a1e18a1384 server add memory profiling 2021-03-30 02:10:53 -07:00
LazyDBA247-Anyvision
9385e2224d fix postgres2 upsertQuery syntax also 2021-03-30 10:51:52 +03:00
Chris Lu
a5662c9e5e adjust ordering 2021-03-30 00:26:57 -07:00
Chris Lu
bec74c3e6a
Merge pull request #1948 from LazyDBA247-Anyvision/master 
add enableUpsert=true
 2021-03-30 00:25:01 -07:00
LazyDBA247-Anyvision
9f1cab179c use a compatible pg syntax for upsert 2021-03-30 10:21:27 +03:00
Chris Lu
a95929e53c reduce default concurrentWriters to 32 2021-03-30 00:17:52 -07:00
LazyDBA247-Anyvision
7f44d953b5 fix GetBool 2021-03-30 01:36:02 +03:00
LazyDBA247-Anyvision
4c51e6a660 add enableUpsert=true 
and rename config to upsertQuery
 2021-03-30 00:32:03 +03:00
Chris Lu
b61e9f6a1a adjust ordering 2021-03-29 00:49:50 -07:00
Chris Lu
1ebdbad222
Merge pull request #1946 from LazyDBA247-Anyvision/master 
Adding Customisation - insertQuery for postgres/mysql
 2021-03-29 00:42:48 -07:00
Konstantin Lebedev
8a95f9c10c iam GetUser 2021-03-29 12:01:44 +05:00
LazyDBA247-Anyvision
96c62bd34d scaffold - insertQuery 
Example for upsert query for mysql & postgres variants
 2021-03-29 10:01:12 +03:00
LazyDBA247-Anyvision
4a02389eb0 Adding custom insertQuery support for postgres/2 mysql/2 2021-03-29 09:58:13 +03:00
Chris Lu
6b7aa9633f 2.36 2021-03-28 19:09:06 -07:00
Chris Lu
bd7471d877 refactor 2021-03-25 12:05:59 -07:00
李海
06be5dc6c3 log snowflake sequencer nodeid's hex when start, it'll be in part of new assigned fid 2021-03-25 18:59:40 +08:00
李海
69b2dab9c6 add a snowflake sequencer as more robust fid generator, but less compressable than small auto-inc id 2021-03-25 18:49:26 +08:00
Konstantin Lebedev
40938d6a47 SaveInsideFiler S3 Configuration 2021-03-25 12:34:11 +05:00
Konstantin Lebedev
9f26f2815c SaveAs S3 Configuration 2021-03-25 12:34:11 +05:00
Konstantin Lebedev
82b0463fac handler PutUserPolicy 
https://docs.aws.amazon.com/IAM/latest/APIReference/API_PutUserPolicy.html
 2021-03-25 12:34:11 +05:00
Konstantin Lebedev
d7719d0542 base handlers 2021-03-25 12:34:11 +05:00
Konstantin Lebedev
03c7953254 init Iam Api Server 2021-03-25 12:34:11 +05:00
Chris Lu
82dfe06066
Merge pull request #1933 from kmlebedev/missingKeysThreshold 
add missingKeysThreshold for fsck
 2021-03-24 10:56:25 -07:00
Konstantin Lebedev
020a5d40c3 avoid counter is null 2021-03-24 22:51:22 +05:00
Konstantin Lebedev
df6cf0a2fa nonRepairThreshold 2021-03-24 22:07:13 +05:00
Konstantin Lebedev
9ce82a9cd2 GOMAXPROCS use defaults to the value of runtime.NumCPU 
https://github.com/chrislusf/seaweedfs/issues/1942
 2021-03-24 18:45:58 +05:00
Konstantin Lebedev
60972f1c97 rename option repairThreshold 2021-03-24 13:24:49 +05:00
Chris Lu
a801332b0d filer: return 409 if file conflicts with a directory 
fix https://github.com/chrislusf/seaweedfs/issues/1938
 2021-03-24 01:03:11 -07:00
Chris Lu
c42b95c596 more help message 
fix https://github.com/chrislusf/seaweedfs/issues/1937
 2021-03-23 17:27:57 -07:00
Konstantin Lebedev
40a9e88c07 rename var 2021-03-23 21:42:35 +05:00
Konstantin Lebedev
d21a5bf5d0 add missingKeysThreshold for fsck 2021-03-23 15:04:07 +05:00
Chris Lu
85cff10787 mysql: ensure name column is case sensitive 
fix https://github.com/chrislusf/seaweedfs/issues/1931
 2021-03-23 00:46:50 -07:00
Chris Lu
4abb511db3 make a local copy of the in memory cached data 2021-03-22 22:33:07 -07:00
Chris Lu
3cbc40fa48 avoid creating multiple reader 2021-03-22 22:32:47 -07:00
Chris Lu
2f7c7afdec refactor: remove unused parameter 2021-03-22 22:13:19 -07:00
Chris Lu
d48dd0c738 use []byte directly instead of bytes.Buffer 2021-03-22 22:12:57 -07:00
Chris Lu
8eb14967ef shell: volume.check.disk adds optional slow mode 
fix https://github.com/chrislusf/seaweedfs/issues/1930
 2021-03-22 21:01:43 -07:00
Chris Lu
288369cfc7 mount: release resources when Forget() is called 
address https://github.com/chrislusf/seaweedfs/issues/1929
 2021-03-22 16:22:19 -07:00
Chris Lu
5d931eff27 avoid possible nil 
fix https://github.com/chrislusf/seaweedfs/issues/1928

The nil was because of `dn.Parent().UnlinkChildNode(dn.Id())` in topo.UnRegisterDataNode() function, when the dn leaves the cluster.
 2021-03-22 13:24:07 -07:00
Chris Lu
f315eb2bb8 fix build error 2021-03-22 00:12:53 -07:00
Chris Lu
1dd5bc134c 2.35 2021-03-22 00:05:09 -07:00
Chris Lu
b465095db1 shell: add volume.check.disk to fix inconsistency for replicated volumes 
fix https://github.com/chrislusf/seaweedfs/issues/1923
 2021-03-22 00:03:16 -07:00
Chris Lu
df461402cc ensure entry attribute is not nill 
fix https://github.com/chrislusf/seaweedfs/issues/1926
 2021-03-21 22:27:30 -07:00
Chris Lu
102a951377 refactor, split into 2 files 2021-03-21 13:05:53 -07:00
Chris Lu
f6e2566ee5 better logs for cookie mismatch 2021-03-20 12:02:57 -07:00
Chris Lu
27e24162f2 minor 2021-03-20 06:37:03 -07:00
Chris Lu
41cc6e661b fs.mv ensures there are 2 arguments 2021-03-20 06:34:13 -07:00
Chris Lu
b2c1c209a5 refactor 2021-03-20 06:34:13 -07:00
Chris Lu
e9b1853d63 go fmt 2021-03-19 01:31:56 -07:00
Chris Lu
b1a86cf808 s3: copy object to itself 
fix https://github.com/chrislusf/seaweedfs/issues/1922
 2021-03-19 01:31:49 -07:00
Konstantin Lebedev
bf94149920 add line 2021-03-17 23:41:34 +05:00
Konstantin Lebedev
c5705e7a35 dir + slash key does not exist 
https://github.com/chrislusf/seaweedfs/issues/1917
 2021-03-17 23:40:42 +05:00
Konstantin Lebedev
eb54993a4e Merge branch 'upstreamMaster' into check_chunkviews_mr 
# Conflicts:
#	weed/filer/filechunk_manifest.go
#	weed/filer/stream.go
#	weed/replication/repl_util/replication_util.go
#	weed/util/fasthttp_util.go
 2021-03-16 15:29:49 +05:00
Chris Lu
9672f9e1b2 2.34 2021-03-16 03:01:15 -07:00
Chris Lu
69694a17be reverting 7d57664c2d 2021-03-16 02:59:26 -07:00
Chris Lu
c00dd5e62e report error if entry not found 2021-03-16 02:55:49 -07:00
Konstantin Lebedev
06da02739d CheckAllChunkViews() for HEAD requests only 2021-03-16 14:15:17 +05:00
Chris Lu
91a3ac9731 2.33 2021-03-16 00:36:06 -07:00
Chris Lu
4b1ed227d1 revert fasthttp changes 
related to https://github.com/chrislusf/seaweedfs/issues/1907
 2021-03-16 00:33:14 -07:00
Chris Lu
10164d0386
Merge pull request #1905 from wuh-fnst/s3ListCounterErr 
s3: count correction for the number of files in the dir
 2021-03-15 20:54:58 -07:00
Chris Lu
22a07a12d2 [mount] [regression] filer.path option results in empty mount in 2.32 
fix https://github.com/chrislusf/seaweedfs/issues/1906
 2021-03-15 20:46:25 -07:00
wuh-fnst
050a7f9599 s3: count correction for the number of files in the dir 2021-03-16 09:22:03 +08:00
Konstantin Lebedev
3a3699867b Status PartialContent for Content-Range response 2021-03-15 23:30:22 +05:00
Konstantin Lebedev
90510e3137 If WriteHeader is not called explicitly, the first call to Write 
will trigger an implicit WriteHeader(http.StatusOK).
WriteHeader: Only one header may be written. Go does not currently!
 2021-03-15 21:22:59 +05:00
Konstantin Lebedev
46b9f5cff4 add debug logging 2021-03-15 19:33:33 +05:00
Konstantin Lebedev
7194a5e7bf avoid http error: superfluous response.WriteHeader 
https://github.com/chrislusf/seaweedfs/issues/1838
 2021-03-15 18:52:59 +05:00
Chris Lu
9f00f95bfb 2.32 2021-03-14 21:32:00 -07:00
Chris Lu
a4cfffc264 shell: fix moving volume, volume server evacuate 
fix https://github.com/chrislusf/seaweedfs/issues/1534
 2021-03-14 21:29:55 -07:00
Chris Lu
ca100568f7 fix parsing 2021-03-14 20:50:14 -07:00
Chris Lu
352ba23f83 revert previous change 
revert 29e62aba00
 2021-03-14 20:49:56 -07:00
Chris Lu
29e62aba00 possible fix for volume balance 
address https://github.com/chrislusf/seaweedfs/issues/1534
 2021-03-14 20:23:19 -07:00
Chris Lu
35d939a1d2 go fmt 2021-03-14 13:21:02 -07:00
Chris Lu
2d4c2db81d filer: leveldb, rocksdb auto create store directory 
fix https://github.com/chrislusf/seaweedfs/issues/1901
 2021-03-14 13:20:14 -07:00
Chris Lu
6d3a96eb56 filer: mysql2, postgres2 trigger actions on bucket creation and deletion 
fix https://github.com/chrislusf/seaweedfs/issues/1877
 2021-03-13 22:07:39 -08:00
Chris Lu
cb423312a4 prevent nil volume message 2021-03-13 11:05:29 -08:00
Chris Lu
972327f966 prevent nil volume nm 2021-03-13 11:04:51 -08:00
Chris Lu
2a68ddb963 default to empty host to bind to tcp4 and tcp6 2021-03-12 15:32:51 -08:00
Chris Lu
5fbcaaf37a fullpath() test is not needed 2021-03-12 09:45:25 -08:00
Chris Lu
7d57664c2d mount: internals switch to filer.Entry instead of protobuf 2021-03-12 00:36:38 -08:00
wuh-fnst
4c1d945e46 make List correctly judge whether it is the last file 2021-03-12 11:22:26 +08:00
Chris Lu
cca66c7fbe print out the url 2021-03-11 18:39:45 -08:00
Chris Lu
7f887c78d2 fix 2021-03-11 18:18:16 -08:00
Chris Lu
fcc52d1e4e shell: add s3.clean.uploads to clean up stale multipart uploads 
fix https://github.com/chrislusf/seaweedfs/issues/1855
 2021-03-11 18:05:24 -08:00
Chris Lu
40dc5ac904 mount: add a bit retry when connecting to filer during startup 2021-03-11 16:36:43 -08:00
Chris Lu
aba47fd9e8 reduce repeated calls to dir.FullPath() 2021-03-11 14:08:20 -08:00
Chris Lu
d084334ffd
Merge pull request #1883 from kmlebedev/passDelNonEmptyFolder 
s3 delete-objects pass OK for fail to delete non-empty folder
 2021-03-11 11:01:44 -08:00
Chris Lu
0e35836cc1
Merge pull request #1890 from kmlebedev/ZeroDataSize 
error if read chunk zero data size
 2021-03-11 10:51:48 -08:00
Konstantin Lebedev
58cdcc6d6e error if urls not found 2021-03-11 23:34:36 +05:00
Chris Lu
79280e1513 escape special characters 
fix https://github.com/chrislusf/seaweedfs/issues/1884
 2021-03-11 09:49:40 -08:00