mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-12-01 07:49:02 +08:00
Code Issues Packages Projects Releases Wiki Activity
5,832 Commits 32 Branches 296 Tags 148 MiB
3668d10664
Commit Graph

3760 Commits

Author SHA1 Message Date
Chris Lu
b971317a16 avoid possible corrupted file names 2021-04-15 11:41:34 -07:00
Chris Lu
283d703d50 adjust text 2021-04-15 11:29:58 -07:00
Chris Lu
609e228578 avoid forward slash in file names 2021-04-15 10:53:04 -07:00
Chris Lu
ba92f2e714 add node.selectedVolumes 
fix https://github.com/chrislusf/seaweedfs/issues/1990
 2021-04-15 03:19:28 -07:00
Chris Lu
16c0304416 ensure to delete on filer also 2021-04-15 02:29:04 -07:00
Chris Lu
217e0f9066 mount: remove folder recursively 2021-04-15 01:51:10 -07:00
Chris Lu
3e669e6d7b mostly refactoring, add some error handling 2021-04-14 23:33:37 -07:00
Chris Lu
07f712c83f fix typo 2021-04-14 23:21:38 -07:00
Chris Lu
36c79de3f4 fuse mount: dir ReadDirAll avoid extra conversion to filer_pb.Entry 2021-04-14 23:21:24 -07:00
Chris Lu
e41766feb6 fuse mount: dir lookup avoids extra conversion to filer_pb.Entry object 2021-04-14 22:38:34 -07:00
Chris Lu
1adc8f86ea lighten up File object 
file.entry only exists when file.isOpen
 2021-04-14 20:49:15 -07:00
Chris Lu
6bc09b18c4 truncate is a bit faster to reuse the storage 2021-04-14 20:26:56 -07:00
Chris Lu
c04b7e106f mount: remove entry from Dir object 2021-04-14 20:26:13 -07:00
Chris Lu
fc0cbf565f add option to obfuscate the file names 2021-04-14 15:37:24 -07:00
Chris Lu
e75633c64f volume.check.disk: break loop for read only volumes 
fix https://github.com/chrislusf/seaweedfs/issues/2002
 2021-04-14 12:40:13 -07:00
Chris Lu
9d50867d08 volume.tier.move: avoid data loss when destination volume server already has the volume 
fix https://github.com/chrislusf/seaweedfs/issues/2001
 2021-04-14 10:26:26 -07:00
Chris Lu
ff4c1d5965 adjust logging 
fix https://github.com/chrislusf/seaweedfs/issues/1999
 2021-04-14 10:04:26 -07:00
qieqieplus
270645f8d7 fix #1996 2021-04-14 18:29:28 +08:00
Chris Lu
90677e1097 ensure to call line.Close() 
fix https://github.com/chrislusf/seaweedfs/issues/1995

similar to https://github.com/peterh/liner/issues/104
 2021-04-14 01:33:21 -07:00
Chris Lu
3f3268cd1b go fmt 2021-04-14 00:30:16 -07:00
Chris Lu
ca0f07a188 move file reader, entryViewCache to file handle 
reduce file object size
 2021-04-14 00:29:58 -07:00
Chris Lu
5985a7d38d add log level possible values 
fix https://github.com/chrislusf/seaweedfs/issues/1989
 2021-04-12 21:15:51 -07:00
Chris Lu
f5de42fae3
Merge pull request #1975 from kmlebedev/iam_handlers 
IAM handlers
 2021-04-12 12:07:45 -07:00
Chris Lu
1e033d45b8 simpler logic 
related to https://github.com/chrislusf/seaweedfs/pull/1981
 2021-04-12 12:04:53 -07:00
Chris Lu
52200a903b
Merge pull request #1981 from ueni-ltd/if-modified-since 
Fix If-Modified-Since behavior
 2021-04-12 12:02:54 -07:00
Chris Lu
519b0e1e49 filer: upload to a directory without "/" suffix 
fix https://github.com/chrislusf/seaweedfs/issues/1988
 2021-04-12 11:56:56 -07:00
Konstantin Lebedev
8e02e138ea Merge branch 'upstreamMaster' into iam_handlers 2021-04-12 11:22:51 +05:00
Chris Lu
742ab1ec81 2.39 2021-04-11 19:47:11 -07:00
Chris Lu
0df5b53ad8 adjust help message 2021-04-11 00:26:28 -07:00
Chris Lu
f62c153274 go fmt 2021-04-10 23:48:18 -07:00
Chris Lu
af313dff58 add gateway for easier POST and DELETE blobs 2021-04-10 23:47:47 -07:00
Konstantin Lebedev
5021bea698 GetUserPolicy 2021-04-10 23:57:45 +05:00
Chris Lu
0b82edc0d2 filer: avoid stuck uploader 
fix https://github.com/chrislusf/seaweedfs/issues/1980

reverting the file upload batch executor
 2021-04-09 13:05:15 -07:00
Chris Lu
93f4146ffa properly release the view cache 2021-04-09 12:36:39 -07:00
Merlin Gaillard
f952f979d1 filer: return 304 when If-Modified-Since == Last-Modified 2021-04-09 15:04:17 +02:00
Merlin Gaillard
4d4acc715e s3api: handle 304 response code from filer 2021-04-09 12:13:19 +02:00
Chris Lu
6deb647a8f mount: fix possible memory leak 
if many files are read repeatedly, their metadata are accumulated in memory. This fix cleared the metadata after the file is read.
 2021-04-08 19:47:31 -07:00
Konstantin Lebedev
ba175f81b5 add auth aws signV4 2021-04-08 17:40:47 +05:00
Konstantin Lebedev
995ae91007 add DeleteUserPolicy 2021-04-08 11:16:36 +05:00
Philippe Pepiot
42a761ee20
Fix typo in weed filer long help 2021-04-07 22:47:23 +02:00
Chris Lu
3be061994f skip connection reset error 
fix https://github.com/chrislusf/seaweedfs/issues/1971

this is because the connections are pooled but the volume server has reset the connection
 2021-04-07 00:54:13 -07:00
Chris Lu
c5b08bac1b remove mac specific mount options 2021-04-07 00:54:13 -07:00
Konstantin Lebedev
f5f8eec8e2 fix get filerGrpcAddress 2021-04-06 13:53:56 +05:00
Konstantin Lebedev
011e6e90ee Merge branch 'upstreamMaster' into iamapipr 2021-04-06 13:50:33 +05:00
Konstantin Lebedev
ed79baa30f add tests 2021-04-06 13:43:08 +05:00
Chris Lu
2327c0756b fix to avoid loop 2021-04-05 23:24:26 -07:00
Chris Lu
a37eca78cd 2.38 2021-04-05 19:41:54 -07:00
Chris Lu
c131764c34 ensure error is sent back in channel 
fix https://github.com/chrislusf/seaweedfs/issues/1966

avoid shared readErr variable
 2021-04-05 19:40:12 -07:00
Chris Lu
0f64f5b9c8 mount: add readOnly option 
fix https://github.com/chrislusf/seaweedfs/issues/1961
 2021-04-04 21:40:58 -07:00
Chris Lu
6eee200c13 2.37 2021-04-04 18:45:48 -07:00
Chris Lu
fbb82a5c9c skip limiting if limit is zero 2021-04-04 18:38:33 -07:00
Chris Lu
8251d1140e refactor 2021-04-04 18:38:33 -07:00
Chris Lu
5dca98fe71 fix typo 2021-04-03 11:37:13 -07:00
Chris Lu
5e64f65632 ensure tables are created 
fix https://github.com/chrislusf/seaweedfs/issues/1957
 2021-04-02 12:02:26 -07:00
Chris Lu
319743d330 go fmt 2021-04-02 02:22:59 -07:00
Chris Lu
8e404a1433 go fmt 2021-04-02 02:22:26 -07:00
Chris Lu
bdf2ddddfd revert to same implementation as before 
This reverts commit 7e8edc3c4a.
 2021-04-02 02:21:38 -07:00
Chris Lu
67e019d54b add missing changes 2021-04-02 01:11:44 -07:00
Chris Lu
7e8edc3c4a refactoring 2021-04-02 01:10:24 -07:00
Chris Lu
cc0df36a9e smaller file chunks are faster 2021-04-01 02:21:59 -07:00
Chris Lu
cefe66f159 dedicated upload processor 
avoid thundering effect of overloading volume servers
 2021-04-01 02:21:40 -07:00
Chris Lu
1f984d2645 refactor buffer pool 2021-04-01 02:20:00 -07:00
Chris Lu
b5880334fc refactor 2021-03-30 21:07:34 -07:00
Chris Lu
e79e2ddeed remove unused MoveEvents 2021-03-30 20:42:44 -07:00
Chris Lu
e06807a55b adjust logs 2021-03-30 20:36:06 -07:00
Chris Lu
ac875976c0 filer, volume: add concurrent upload size limit to avoid OOM 
add some back pressure when writes are slow
 2021-03-30 02:10:53 -07:00
Chris Lu
a1e18a1384 server add memory profiling 2021-03-30 02:10:53 -07:00
LazyDBA247-Anyvision
9385e2224d fix postgres2 upsertQuery syntax also 2021-03-30 10:51:52 +03:00
Chris Lu
a5662c9e5e adjust ordering 2021-03-30 00:26:57 -07:00
Chris Lu
bec74c3e6a
Merge pull request #1948 from LazyDBA247-Anyvision/master 
add enableUpsert=true
 2021-03-30 00:25:01 -07:00
LazyDBA247-Anyvision
9f1cab179c use a compatible pg syntax for upsert 2021-03-30 10:21:27 +03:00
Chris Lu
a95929e53c reduce default concurrentWriters to 32 2021-03-30 00:17:52 -07:00
LazyDBA247-Anyvision
7f44d953b5 fix GetBool 2021-03-30 01:36:02 +03:00
LazyDBA247-Anyvision
4c51e6a660 add enableUpsert=true 
and rename config to upsertQuery
 2021-03-30 00:32:03 +03:00
Chris Lu
b61e9f6a1a adjust ordering 2021-03-29 00:49:50 -07:00
Chris Lu
1ebdbad222
Merge pull request #1946 from LazyDBA247-Anyvision/master 
Adding Customisation - insertQuery for postgres/mysql
 2021-03-29 00:42:48 -07:00
Konstantin Lebedev
8a95f9c10c iam GetUser 2021-03-29 12:01:44 +05:00
LazyDBA247-Anyvision
96c62bd34d scaffold - insertQuery 
Example for upsert query for mysql & postgres variants
 2021-03-29 10:01:12 +03:00
LazyDBA247-Anyvision
4a02389eb0 Adding custom insertQuery support for postgres/2 mysql/2 2021-03-29 09:58:13 +03:00
Chris Lu
6b7aa9633f 2.36 2021-03-28 19:09:06 -07:00
Chris Lu
bd7471d877 refactor 2021-03-25 12:05:59 -07:00
李海
06be5dc6c3 log snowflake sequencer nodeid's hex when start, it'll be in part of new assigned fid 2021-03-25 18:59:40 +08:00
李海
69b2dab9c6 add a snowflake sequencer as more robust fid generator, but less compressable than small auto-inc id 2021-03-25 18:49:26 +08:00
Konstantin Lebedev
40938d6a47 SaveInsideFiler S3 Configuration 2021-03-25 12:34:11 +05:00
Konstantin Lebedev
9f26f2815c SaveAs S3 Configuration 2021-03-25 12:34:11 +05:00
Konstantin Lebedev
82b0463fac handler PutUserPolicy 
https://docs.aws.amazon.com/IAM/latest/APIReference/API_PutUserPolicy.html
 2021-03-25 12:34:11 +05:00
Konstantin Lebedev
d7719d0542 base handlers 2021-03-25 12:34:11 +05:00
Konstantin Lebedev
03c7953254 init Iam Api Server 2021-03-25 12:34:11 +05:00
Chris Lu
82dfe06066
Merge pull request #1933 from kmlebedev/missingKeysThreshold 
add missingKeysThreshold for fsck
 2021-03-24 10:56:25 -07:00
Konstantin Lebedev
020a5d40c3 avoid counter is null 2021-03-24 22:51:22 +05:00
Konstantin Lebedev
df6cf0a2fa nonRepairThreshold 2021-03-24 22:07:13 +05:00
Konstantin Lebedev
9ce82a9cd2 GOMAXPROCS use defaults to the value of runtime.NumCPU 
https://github.com/chrislusf/seaweedfs/issues/1942
 2021-03-24 18:45:58 +05:00
Konstantin Lebedev
60972f1c97 rename option repairThreshold 2021-03-24 13:24:49 +05:00
Chris Lu
a801332b0d filer: return 409 if file conflicts with a directory 
fix https://github.com/chrislusf/seaweedfs/issues/1938
 2021-03-24 01:03:11 -07:00
Chris Lu
c42b95c596 more help message 
fix https://github.com/chrislusf/seaweedfs/issues/1937
 2021-03-23 17:27:57 -07:00
Konstantin Lebedev
40a9e88c07 rename var 2021-03-23 21:42:35 +05:00
Konstantin Lebedev
d21a5bf5d0 add missingKeysThreshold for fsck 2021-03-23 15:04:07 +05:00
Chris Lu
85cff10787 mysql: ensure name column is case sensitive 
fix https://github.com/chrislusf/seaweedfs/issues/1931
 2021-03-23 00:46:50 -07:00
Chris Lu
4abb511db3 make a local copy of the in memory cached data 2021-03-22 22:33:07 -07:00
Chris Lu
3cbc40fa48 avoid creating multiple reader 2021-03-22 22:32:47 -07:00
Chris Lu
2f7c7afdec refactor: remove unused parameter 2021-03-22 22:13:19 -07:00
Chris Lu
d48dd0c738 use []byte directly instead of bytes.Buffer 2021-03-22 22:12:57 -07:00
Chris Lu
8eb14967ef shell: volume.check.disk adds optional slow mode 
fix https://github.com/chrislusf/seaweedfs/issues/1930
 2021-03-22 21:01:43 -07:00
Chris Lu
288369cfc7 mount: release resources when Forget() is called 
address https://github.com/chrislusf/seaweedfs/issues/1929
 2021-03-22 16:22:19 -07:00
Chris Lu
5d931eff27 avoid possible nil 
fix https://github.com/chrislusf/seaweedfs/issues/1928

The nil was because of `dn.Parent().UnlinkChildNode(dn.Id())` in topo.UnRegisterDataNode() function, when the dn leaves the cluster.
 2021-03-22 13:24:07 -07:00
Chris Lu
f315eb2bb8 fix build error 2021-03-22 00:12:53 -07:00
Chris Lu
1dd5bc134c 2.35 2021-03-22 00:05:09 -07:00
Chris Lu
b465095db1 shell: add volume.check.disk to fix inconsistency for replicated volumes 
fix https://github.com/chrislusf/seaweedfs/issues/1923
 2021-03-22 00:03:16 -07:00
Chris Lu
df461402cc ensure entry attribute is not nill 
fix https://github.com/chrislusf/seaweedfs/issues/1926
 2021-03-21 22:27:30 -07:00
Chris Lu
102a951377 refactor, split into 2 files 2021-03-21 13:05:53 -07:00
Chris Lu
f6e2566ee5 better logs for cookie mismatch 2021-03-20 12:02:57 -07:00
Chris Lu
27e24162f2 minor 2021-03-20 06:37:03 -07:00
Chris Lu
41cc6e661b fs.mv ensures there are 2 arguments 2021-03-20 06:34:13 -07:00
Chris Lu
b2c1c209a5 refactor 2021-03-20 06:34:13 -07:00
Chris Lu
e9b1853d63 go fmt 2021-03-19 01:31:56 -07:00
Chris Lu
b1a86cf808 s3: copy object to itself 
fix https://github.com/chrislusf/seaweedfs/issues/1922
 2021-03-19 01:31:49 -07:00
Konstantin Lebedev
bf94149920 add line 2021-03-17 23:41:34 +05:00
Konstantin Lebedev
c5705e7a35 dir + slash key does not exist 
https://github.com/chrislusf/seaweedfs/issues/1917
 2021-03-17 23:40:42 +05:00
Konstantin Lebedev
eb54993a4e Merge branch 'upstreamMaster' into check_chunkviews_mr 
# Conflicts:
#	weed/filer/filechunk_manifest.go
#	weed/filer/stream.go
#	weed/replication/repl_util/replication_util.go
#	weed/util/fasthttp_util.go
 2021-03-16 15:29:49 +05:00
Chris Lu
9672f9e1b2 2.34 2021-03-16 03:01:15 -07:00
Chris Lu
69694a17be reverting 7d57664c2d 2021-03-16 02:59:26 -07:00
Chris Lu
c00dd5e62e report error if entry not found 2021-03-16 02:55:49 -07:00
Konstantin Lebedev
06da02739d CheckAllChunkViews() for HEAD requests only 2021-03-16 14:15:17 +05:00
Chris Lu
91a3ac9731 2.33 2021-03-16 00:36:06 -07:00
Chris Lu
4b1ed227d1 revert fasthttp changes 
related to https://github.com/chrislusf/seaweedfs/issues/1907
 2021-03-16 00:33:14 -07:00
Chris Lu
10164d0386
Merge pull request #1905 from wuh-fnst/s3ListCounterErr 
s3: count correction for the number of files in the dir
 2021-03-15 20:54:58 -07:00
Chris Lu
22a07a12d2 [mount] [regression] filer.path option results in empty mount in 2.32 
fix https://github.com/chrislusf/seaweedfs/issues/1906
 2021-03-15 20:46:25 -07:00
wuh-fnst
050a7f9599 s3: count correction for the number of files in the dir 2021-03-16 09:22:03 +08:00
Konstantin Lebedev
3a3699867b Status PartialContent for Content-Range response 2021-03-15 23:30:22 +05:00
Konstantin Lebedev
90510e3137 If WriteHeader is not called explicitly, the first call to Write 
will trigger an implicit WriteHeader(http.StatusOK).
WriteHeader: Only one header may be written. Go does not currently!
 2021-03-15 21:22:59 +05:00
Konstantin Lebedev
46b9f5cff4 add debug logging 2021-03-15 19:33:33 +05:00
Konstantin Lebedev
7194a5e7bf avoid http error: superfluous response.WriteHeader 
https://github.com/chrislusf/seaweedfs/issues/1838
 2021-03-15 18:52:59 +05:00
Chris Lu
9f00f95bfb 2.32 2021-03-14 21:32:00 -07:00
Chris Lu
a4cfffc264 shell: fix moving volume, volume server evacuate 
fix https://github.com/chrislusf/seaweedfs/issues/1534
 2021-03-14 21:29:55 -07:00
Chris Lu
ca100568f7 fix parsing 2021-03-14 20:50:14 -07:00
Chris Lu
352ba23f83 revert previous change 
revert 29e62aba00
 2021-03-14 20:49:56 -07:00
Chris Lu
29e62aba00 possible fix for volume balance 
address https://github.com/chrislusf/seaweedfs/issues/1534
 2021-03-14 20:23:19 -07:00
Chris Lu
35d939a1d2 go fmt 2021-03-14 13:21:02 -07:00
Chris Lu
2d4c2db81d filer: leveldb, rocksdb auto create store directory 
fix https://github.com/chrislusf/seaweedfs/issues/1901
 2021-03-14 13:20:14 -07:00
Chris Lu
6d3a96eb56 filer: mysql2, postgres2 trigger actions on bucket creation and deletion 
fix https://github.com/chrislusf/seaweedfs/issues/1877
 2021-03-13 22:07:39 -08:00
Chris Lu
cb423312a4 prevent nil volume message 2021-03-13 11:05:29 -08:00
Chris Lu
972327f966 prevent nil volume nm 2021-03-13 11:04:51 -08:00
Chris Lu
2a68ddb963 default to empty host to bind to tcp4 and tcp6 2021-03-12 15:32:51 -08:00
Chris Lu
5fbcaaf37a fullpath() test is not needed 2021-03-12 09:45:25 -08:00
Chris Lu
7d57664c2d mount: internals switch to filer.Entry instead of protobuf 2021-03-12 00:36:38 -08:00
wuh-fnst
4c1d945e46 make List correctly judge whether it is the last file 2021-03-12 11:22:26 +08:00
Chris Lu
cca66c7fbe print out the url 2021-03-11 18:39:45 -08:00
Chris Lu
7f887c78d2 fix 2021-03-11 18:18:16 -08:00
Chris Lu
fcc52d1e4e shell: add s3.clean.uploads to clean up stale multipart uploads 
fix https://github.com/chrislusf/seaweedfs/issues/1855
 2021-03-11 18:05:24 -08:00
Chris Lu
40dc5ac904 mount: add a bit retry when connecting to filer during startup 2021-03-11 16:36:43 -08:00
Chris Lu
aba47fd9e8 reduce repeated calls to dir.FullPath() 2021-03-11 14:08:20 -08:00
Chris Lu
d084334ffd
Merge pull request #1883 from kmlebedev/passDelNonEmptyFolder 
s3 delete-objects pass OK for fail to delete non-empty folder
 2021-03-11 11:01:44 -08:00
Chris Lu
0e35836cc1
Merge pull request #1890 from kmlebedev/ZeroDataSize 
error if read chunk zero data size
 2021-03-11 10:51:48 -08:00
Konstantin Lebedev
58cdcc6d6e error if urls not found 2021-03-11 23:34:36 +05:00
Chris Lu
79280e1513 escape special characters 
fix https://github.com/chrislusf/seaweedfs/issues/1884
 2021-03-11 09:49:40 -08:00
Konstantin Lebedev
4db21012c1 error if read chunk zero data size 2021-03-11 22:38:59 +05:00
wuheng
828fbf3fb7 s3: "isLast" returns true when the file does not exist 2021-03-11 15:20:50 +08:00
Chris Lu
5d446673a6 fix error printing 2021-03-10 17:13:04 -08:00
Chris Lu
ca546f47e1 s3: escape object key if containing special characters 
fix https://github.com/chrislusf/seaweedfs/issues/1884
 2021-03-10 13:19:28 -08:00
Chris Lu
105fcd7bfc
Merge pull request #1872 from kmlebedev/grpc_auth 
TLS allowed CommonNames
 2021-03-10 11:17:49 -08:00
Konstantin Lebedev
bb3af2d70b no directoriesWithDeletion 2021-03-10 19:52:41 +05:00
Konstantin Lebedev
15b018da34 s3 delete-objects pass OK for fail to delete non-empty folder 2021-03-10 19:41:35 +05:00
Konstantin Lebedev
348e21a08c add comments 2021-03-10 14:42:39 +05:00
Konstantin Lebedev
831953c55c allowed wildcard domain 2021-03-10 14:02:13 +05:00
Konstantin Lebedev
4bf93d6e63 comma-separated 2021-03-10 12:43:13 +05:00
Konstantin Lebedev
0e02f7e258 comma-separated SSL certificate common names 2021-03-10 12:42:44 +05:00
Chris Lu
737bde5ab7 refactoring, remove duplicated code 2021-03-09 23:23:01 -08:00
Chris Lu
5ba4b479f8 properly lock file.entry object 
fix https://github.com/chrislusf/seaweedfs/issues/1882
 2021-03-09 23:08:38 -08:00
Chris Lu
5be83dd4a9 add comments 2021-03-09 14:13:48 -08:00
Chris Lu
9d8ca513d9 refactor 2021-03-09 13:21:26 -08:00
Chris Lu
f5b5e4da2d 2.31 2021-03-09 12:52:16 -08:00
Chris Lu
828f6e9f4d volume: auto add missing vif files 
fix https://github.com/chrislusf/seaweedfs/issues/1878
 2021-03-09 12:09:32 -08:00
Chris Lu
387c6f4218 Revert "delete the folder object in multi_object_delete" 
This reverts commit 2e89c8c9ae.
 2021-03-09 10:07:27 -08:00
qieqieplus
5b16820924 fix: collection stats won't update if all volumes expired at same iteration 2021-03-09 18:49:45 +08:00
Konstantin Lebedev
190fada1ef TLS allowed commonNames 2021-03-08 21:39:44 +05:00
Konstantin Lebedev
c6d3735605 permitCommonNames 
https://github.com/chrislusf/seaweedfs/issues/1841
https://jbrandhorst.com/post/grpc-auth/
 2021-03-08 13:16:17 +05:00
Chris Lu
d888e9a90f 2.30 2021-03-07 15:01:19 -08:00
Chris Lu
60686a0bb7 make tcp optional 2021-03-07 14:45:36 -08:00
Chris Lu
3b2a9c98ef fix nil 2021-03-07 11:45:35 -08:00
Chris Lu
df72dc206d stats collect chunk upload retry count 2021-03-07 11:26:15 -08:00
Chris Lu
726edab054 avoid nil when closing an index 
fix https://github.com/chrislusf/seaweedfs/issues/1870
 2021-03-07 11:03:09 -08:00
Chris Lu
bdfed16d42 avoid nil exception 
fix https://github.com/chrislusf/seaweedfs/issues/1869
 2021-03-07 01:49:06 -08:00
Chris Lu
1bd880dcdb adds tcp writes benchmark 2021-03-06 14:26:27 -08:00
Chris Lu
9d402ebe9f refactoring 2021-03-06 14:26:27 -08:00
Chris Lu
e9eddfb8a2 remove wrong writes 2021-03-06 14:26:27 -08:00
Chris Lu
8363be8548 trim out trailing return character 2021-03-06 14:26:27 -08:00
Chris Lu
38fc200e56 CRCWriter consistent with CRC 2021-03-06 14:26:27 -08:00
Chris Lu
1444e9d275 migrated multi host connection pool from godropbox package 
removing unneeded dependencies, which involved etcd versions.
 2021-03-06 14:26:27 -08:00
Patrick Schmidt
7413d59750 Fix EC shard count logic 
This fixes the calculation of the amount of EC shards a node holds.
Previously a global counter was increased, but also used inside the
loop to apply disk usage deltas. This led to wrong absolute numbers.
The fix is to apply only deltas of single EC shards per iteration.
 2021-03-05 12:50:58 +01:00
Chris Lu
400de380f4 volume server: support tcp direct put/get/delete 2021-03-05 02:29:38 -08:00
Chris Lu
2e89c8c9ae delete the folder object in multi_object_delete 
fix https://github.com/chrislusf/seaweedfs/issues/1857
 2021-03-04 14:03:40 -08:00
Chris Lu
c5cb346a08 fix compilation error 2021-03-04 12:36:20 -08:00
Chris Lu
f5cc96f289 skip checking master peers if not starting master 2021-03-04 12:31:26 -08:00
Chris Lu
40ff30b83f adjust help message 2021-03-03 02:12:46 -08:00
Chris Lu
c0842fe99f add filer.meta.backup command 2021-03-03 02:02:29 -08:00
Chris Lu
0c5c51eb98 adjust help message 2021-03-02 23:07:39 -08:00
Chris Lu
bd727b7b53 go fmt 2021-03-02 20:59:56 -08:00
Chris Lu
e982b06bcd support IPv6 2021-03-02 20:59:39 -08:00
Chris Lu
bcf32591b7 reduce possibility of nil entry 2021-03-02 13:33:56 -08:00
Chris Lu
be9c7c21ec reduce possibility of nil file entry 2021-03-02 11:17:17 -08:00
Chris Lu
02e146f3e1 return err 2021-03-02 08:54:18 -08:00
Chris Lu
5511722420 s3: list permission is needed to list my buckets 
fix https://github.com/chrislusf/seaweedfs/issues/1837
 2021-03-01 12:41:55 -08:00
Chris Lu
c46d16b2f3 skip network error when remote volumes start 2021-03-01 01:20:06 -08:00
Chris Lu
6e43e8ce17 add logs for remote file loading 2021-03-01 00:48:59 -08:00
Chris Lu
015d16f43f add vif file versions in case loading superblock fails 2021-03-01 00:48:30 -08:00
Chris Lu
d680676d45 skip already loaded backends 2021-03-01 00:47:03 -08:00
Chris Lu
540441fd38 go fmt 2021-02-28 20:34:14 -08:00
Chris Lu
c9722dceb2 adjust log level 2021-02-28 20:26:55 -08:00
Chris Lu
1b1c018165 adjust text 2021-02-28 19:02:43 -08:00
Chris Lu
2cca07b44c cloud tier to non-AWS s3 gateways 2021-02-28 18:59:09 -08:00
Chris Lu
f29ae0db76 2.29 2021-02-28 18:08:26 -08:00
bingoohuang
5cdff56731 set default env prefix to WEED_ 2021-03-01 09:35:23 +08:00
Chris Lu
e52c94640e filer.backup: added to replace filer.replicate 2021-02-28 16:22:27 -08:00
Chris Lu
014a31d11a minor 2021-02-28 16:21:09 -08:00
Chris Lu
f2fcb77808 local file sink: create backup files with permission 0755 2021-02-28 16:20:47 -08:00
Chris Lu
984fdd6192 always use non bucket prefixing url 2021-02-28 16:20:13 -08:00
Chris Lu
9abb041763 filer source: support filerProxy mode 2021-02-28 16:19:47 -08:00
Chris Lu
678c54d705 data sink: add incremental mode 2021-02-28 16:19:03 -08:00
Chris Lu
4ff2c5c4c9 rename file 2021-02-28 16:14:21 -08:00
Chris Lu
3b76a51f5f add help message 2021-02-27 12:15:49 -08:00
Chris Lu
70434df105 use "options" 2021-02-27 12:12:53 -08:00
Chris Lu
d3c31c69a7 avoid confusion: conf and config are too similar 2021-02-27 12:08:09 -08:00
Chris Lu
543c8daa6d
Merge pull request #1844 from kmlebedev/processRangeRequest 2021-02-26 09:18:22 -08:00
Patrick Schmidt
5f7b024891 Show the real disk usage in stats calls 
Currently the file size of only one volume location is taken into
account in the stats. This commit multiplies the disk usages by the
amount of nodes holding a replica of the volume.
This will yield the expected amount of disk usage and matches the
total size calculations from before.
 2021-02-26 13:58:40 +01:00
Konstantin Lebedev
ee21c0042e log error write entry 2021-02-26 14:18:01 +05:00
Chris Lu
7ba75e3d5a filer: do not return no content for empty files 
fix https://github.com/chrislusf/seaweedfs/issues/1831
fix https://github.com/chrislusf/seaweedfs/issues/1830
 2021-02-23 12:25:27 -08:00
Chris Lu
9edd964627 volume.tier.move: avoid repeated move for replicated volumes 
fix https://github.com/chrislusf/seaweedfs/issues/1792#issuecomment-784139348
 2021-02-23 03:49:14 -08:00
Chris Lu
37f104f88f 2.28 2021-02-22 22:54:34 -08:00
Chris Lu
90cdf9dcac avoid conflict with "weed scaffold -config=xxx" 2021-02-22 16:57:27 -08:00
Chris Lu
72b0d9d8c4 avoid unnecessary user home checking 2021-02-22 16:50:56 -08:00
Chris Lu
2270737344 volume: avoid fixed vacuum timeout for large volumes 
1GB for 3 minutes, about 5.7MB/s
 2021-02-22 12:52:37 -08:00
Chris Lu
44bdfb2d15 filer: avoid encryption and compression at the same time 
fix https://github.com/chrislusf/seaweedfs/issues/1828
 2021-02-22 12:22:49 -08:00
Chris Lu
62191b08ea disk type support custom tags 2021-02-22 02:03:12 -08:00
Chris Lu
30b30b8fe0 volume.tier.move: passing non-empty disk type 2021-02-22 01:59:03 -08:00
Chris Lu
5da63e045e avoid moving to another server with the same volume id 2021-02-22 01:44:18 -08:00
Chris Lu
6a4546d2c0 shell: add volume.tier.move 2021-02-22 01:30:07 -08:00
Chris Lu
1c233ad986 refactoring 2021-02-22 00:28:42 -08:00
Chris Lu
151c281f36 2.27 2021-02-21 19:29:27 -08:00
Chris Lu
258e93bc86
Merge pull request #1823 from bingoohuang/master 
feature: support command line arguments in a configuration file
 2021-02-20 22:35:18 -08:00
Chris Lu
03c643aa6b fix test 
fileSize need to be divided by 8
 2021-02-20 21:19:21 -08:00
Chris Lu
7635f6b9fa disk file avoid file.Stat() 2021-02-20 20:06:06 -08:00
bingoo
1af6c96cd0 fix log help 2021-02-21 10:46:01 +08:00
Chris Lu
a2383b3b12 Revert "Revert "avoid file.Stat()"" 
This reverts commit 7ef4c24f18.
 2021-02-20 12:42:09 -08:00
Chris Lu
7ef4c24f18 Revert "avoid file.Stat()" 
This reverts commit 98c93ca465.
 2021-02-20 12:39:33 -08:00
Chris Lu
1ad3200094 skip seek() when index file writes 2021-02-20 12:39:25 -08:00
Chris Lu
98c93ca465 avoid file.Stat() 
avoid one Syscall, but did not help on performance though
 2021-02-20 09:44:17 -08:00
bingoohuang
ee082ae402 revert comment temporarily 2021-02-20 16:59:38 +08:00
bingoohuang
ee7cdf3668 revert ErrXyz to ErrorXyz temporarily. 2021-02-20 16:58:08 +08:00
bingoohuang
50df484d86 support command line arguments in a configuration file so we can weed master -config=master.conf, weed volume -config=volume.conf and etc. 2021-02-20 16:52:57 +08:00
bingoohuang
f69356f589
Merge pull request #3 from chrislusf/master 2021-02-20 16:45:02 +08:00