mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-12-15 11:09:06 +08:00
Code Issues Packages Projects Releases Wiki Activity
6,957 Commits 32 Branches 296 Tags 150 MiB
121b31f750
Commit Graph

4629 Commits

Author SHA1 Message Date
Chris Lu
ca998328c2 do not add new volumes when below minFreeSpacePercent 
fix https://github.com/chrislusf/seaweedfs/issues/2017
 2021-04-21 23:11:11 -07:00
Chris Lu
cd7bf1a72a filer.copy copy empty folders 
fix https://github.com/chrislusf/seaweedfs/issues/2016
 2021-04-21 11:17:43 -07:00
Chris Lu
a8114da02d avoid thundering herd effect 
transient errors may cause thundering herd effect to all trying to recover from remove ec shards
 2021-04-21 10:17:12 -07:00
Chris Lu
ae74d8f02a fix error message 
related to https://github.com/chrislusf/seaweedfs/issues/2012
 2021-04-21 01:40:16 -07:00
Chris Lu
11c405fc85 ensure file handles are released 2021-04-20 19:56:55 -07:00
liuxiaobo
c31c5e829c
fix path-specific filer store comment error 2021-04-20 10:08:58 +08:00
Chris Lu
83cf94ad2d delay new file creation unless file is opened exclusively 2021-04-19 10:58:25 -07:00
Chris Lu
e983f91b03 2.40 2021-04-18 13:58:01 -07:00
Chris Lu
d1c813c470 let the fuse library manage directory id 
otherwise, on mac, during large directory deletion, if some ReaDirAll happens, the lib seems confused about the directories, and some child directories are not deleted.
 2021-04-18 13:07:28 -07:00
Chris Lu
372872ebbf set root node inode number 2021-04-18 13:07:28 -07:00
Chris Lu
e332da4837 set inode value 2021-04-18 13:07:28 -07:00
Chris Lu
2acf6be24e resend the http request if connection is stale 2021-04-18 13:07:28 -07:00
Chris Lu
6cbd786db9 correctly runs git clone 2021-04-18 13:07:28 -07:00
Chris Lu
d41e6826d3 adjust logging 2021-04-18 13:06:38 -07:00
Chris Lu
d9a2a7f1c4 WIP 
no memory issue

if some directory is removed, it may have this error

$ rm -Rf ~/tmp/m2/s1
rm: fts_read: Device not configured
 2021-04-18 13:06:38 -07:00
Chris Lu
54410ca955 cleaner way to set readonly 2021-04-18 10:02:02 -07:00
Konstantin Lebedev
198688c717 revert volume etag 2021-04-16 23:22:31 +05:00
Chris Lu
c83ab91e2e remove unused variable 2021-04-16 10:34:02 -07:00
Chris Lu
3074e9b428 ensure consistent inode value 2021-04-15 22:42:24 -07:00
Chris Lu
b971317a16 avoid possible corrupted file names 2021-04-15 11:41:34 -07:00
Chris Lu
283d703d50 adjust text 2021-04-15 11:29:58 -07:00
Chris Lu
609e228578 avoid forward slash in file names 2021-04-15 10:53:04 -07:00
Chris Lu
ba92f2e714 add node.selectedVolumes 
fix https://github.com/chrislusf/seaweedfs/issues/1990
 2021-04-15 03:19:28 -07:00
Chris Lu
16c0304416 ensure to delete on filer also 2021-04-15 02:29:04 -07:00
Chris Lu
217e0f9066 mount: remove folder recursively 2021-04-15 01:51:10 -07:00
Chris Lu
3e669e6d7b mostly refactoring, add some error handling 2021-04-14 23:33:37 -07:00
Chris Lu
07f712c83f fix typo 2021-04-14 23:21:38 -07:00
Chris Lu
36c79de3f4 fuse mount: dir ReadDirAll avoid extra conversion to filer_pb.Entry 2021-04-14 23:21:24 -07:00
Chris Lu
e41766feb6 fuse mount: dir lookup avoids extra conversion to filer_pb.Entry object 2021-04-14 22:38:34 -07:00
Chris Lu
1adc8f86ea lighten up File object 
file.entry only exists when file.isOpen
 2021-04-14 20:49:15 -07:00
Chris Lu
6bc09b18c4 truncate is a bit faster to reuse the storage 2021-04-14 20:26:56 -07:00
Chris Lu
c04b7e106f mount: remove entry from Dir object 2021-04-14 20:26:13 -07:00
Chris Lu
fc0cbf565f add option to obfuscate the file names 2021-04-14 15:37:24 -07:00
Chris Lu
e75633c64f volume.check.disk: break loop for read only volumes 
fix https://github.com/chrislusf/seaweedfs/issues/2002
 2021-04-14 12:40:13 -07:00
Chris Lu
9d50867d08 volume.tier.move: avoid data loss when destination volume server already has the volume 
fix https://github.com/chrislusf/seaweedfs/issues/2001
 2021-04-14 10:26:26 -07:00
Chris Lu
ff4c1d5965 adjust logging 
fix https://github.com/chrislusf/seaweedfs/issues/1999
 2021-04-14 10:04:26 -07:00
qieqieplus
270645f8d7 fix #1996 2021-04-14 18:29:28 +08:00
Chris Lu
90677e1097 ensure to call line.Close() 
fix https://github.com/chrislusf/seaweedfs/issues/1995

similar to https://github.com/peterh/liner/issues/104
 2021-04-14 01:33:21 -07:00
Chris Lu
3f3268cd1b go fmt 2021-04-14 00:30:16 -07:00
Chris Lu
ca0f07a188 move file reader, entryViewCache to file handle 
reduce file object size
 2021-04-14 00:29:58 -07:00
Chris Lu
5985a7d38d add log level possible values 
fix https://github.com/chrislusf/seaweedfs/issues/1989
 2021-04-12 21:15:51 -07:00
Chris Lu
f5de42fae3
Merge pull request #1975 from kmlebedev/iam_handlers 
IAM handlers
 2021-04-12 12:07:45 -07:00
Chris Lu
1e033d45b8 simpler logic 
related to https://github.com/chrislusf/seaweedfs/pull/1981
 2021-04-12 12:04:53 -07:00
Chris Lu
52200a903b
Merge pull request #1981 from ueni-ltd/if-modified-since 
Fix If-Modified-Since behavior
 2021-04-12 12:02:54 -07:00
Chris Lu
519b0e1e49 filer: upload to a directory without "/" suffix 
fix https://github.com/chrislusf/seaweedfs/issues/1988
 2021-04-12 11:56:56 -07:00
Konstantin Lebedev
8e02e138ea Merge branch 'upstreamMaster' into iam_handlers 2021-04-12 11:22:51 +05:00
Chris Lu
742ab1ec81 2.39 2021-04-11 19:47:11 -07:00
Chris Lu
0df5b53ad8 adjust help message 2021-04-11 00:26:28 -07:00
Chris Lu
f62c153274 go fmt 2021-04-10 23:48:18 -07:00
Chris Lu
af313dff58 add gateway for easier POST and DELETE blobs 2021-04-10 23:47:47 -07:00
Konstantin Lebedev
5021bea698 GetUserPolicy 2021-04-10 23:57:45 +05:00
Chris Lu
0b82edc0d2 filer: avoid stuck uploader 
fix https://github.com/chrislusf/seaweedfs/issues/1980

reverting the file upload batch executor
 2021-04-09 13:05:15 -07:00
Chris Lu
93f4146ffa properly release the view cache 2021-04-09 12:36:39 -07:00
Merlin Gaillard
f952f979d1 filer: return 304 when If-Modified-Since == Last-Modified 2021-04-09 15:04:17 +02:00
Merlin Gaillard
4d4acc715e s3api: handle 304 response code from filer 2021-04-09 12:13:19 +02:00
Chris Lu
6deb647a8f mount: fix possible memory leak 
if many files are read repeatedly, their metadata are accumulated in memory. This fix cleared the metadata after the file is read.
 2021-04-08 19:47:31 -07:00
Konstantin Lebedev
ba175f81b5 add auth aws signV4 2021-04-08 17:40:47 +05:00
Konstantin Lebedev
995ae91007 add DeleteUserPolicy 2021-04-08 11:16:36 +05:00
Philippe Pepiot
42a761ee20
Fix typo in weed filer long help 2021-04-07 22:47:23 +02:00
Chris Lu
3be061994f skip connection reset error 
fix https://github.com/chrislusf/seaweedfs/issues/1971

this is because the connections are pooled but the volume server has reset the connection
 2021-04-07 00:54:13 -07:00
Chris Lu
c5b08bac1b remove mac specific mount options 2021-04-07 00:54:13 -07:00
Konstantin Lebedev
f5f8eec8e2 fix get filerGrpcAddress 2021-04-06 13:53:56 +05:00
Konstantin Lebedev
011e6e90ee Merge branch 'upstreamMaster' into iamapipr 2021-04-06 13:50:33 +05:00
Konstantin Lebedev
ed79baa30f add tests 2021-04-06 13:43:08 +05:00
Chris Lu
2327c0756b fix to avoid loop 2021-04-05 23:24:26 -07:00
Chris Lu
a37eca78cd 2.38 2021-04-05 19:41:54 -07:00
Chris Lu
c131764c34 ensure error is sent back in channel 
fix https://github.com/chrislusf/seaweedfs/issues/1966

avoid shared readErr variable
 2021-04-05 19:40:12 -07:00
Chris Lu
0f64f5b9c8 mount: add readOnly option 
fix https://github.com/chrislusf/seaweedfs/issues/1961
 2021-04-04 21:40:58 -07:00
Chris Lu
6eee200c13 2.37 2021-04-04 18:45:48 -07:00
Chris Lu
fbb82a5c9c skip limiting if limit is zero 2021-04-04 18:38:33 -07:00
Chris Lu
8251d1140e refactor 2021-04-04 18:38:33 -07:00
Chris Lu
5dca98fe71 fix typo 2021-04-03 11:37:13 -07:00
Chris Lu
5e64f65632 ensure tables are created 
fix https://github.com/chrislusf/seaweedfs/issues/1957
 2021-04-02 12:02:26 -07:00
Chris Lu
319743d330 go fmt 2021-04-02 02:22:59 -07:00
Chris Lu
8e404a1433 go fmt 2021-04-02 02:22:26 -07:00
Chris Lu
bdf2ddddfd revert to same implementation as before 
This reverts commit 7e8edc3c4a.
 2021-04-02 02:21:38 -07:00
Chris Lu
67e019d54b add missing changes 2021-04-02 01:11:44 -07:00
Chris Lu
7e8edc3c4a refactoring 2021-04-02 01:10:24 -07:00
Chris Lu
cc0df36a9e smaller file chunks are faster 2021-04-01 02:21:59 -07:00
Chris Lu
cefe66f159 dedicated upload processor 
avoid thundering effect of overloading volume servers
 2021-04-01 02:21:40 -07:00
Chris Lu
1f984d2645 refactor buffer pool 2021-04-01 02:20:00 -07:00
Chris Lu
b5880334fc refactor 2021-03-30 21:07:34 -07:00
Chris Lu
e79e2ddeed remove unused MoveEvents 2021-03-30 20:42:44 -07:00
Chris Lu
e06807a55b adjust logs 2021-03-30 20:36:06 -07:00
Chris Lu
ac875976c0 filer, volume: add concurrent upload size limit to avoid OOM 
add some back pressure when writes are slow
 2021-03-30 02:10:53 -07:00
Chris Lu
a1e18a1384 server add memory profiling 2021-03-30 02:10:53 -07:00
LazyDBA247-Anyvision
9385e2224d fix postgres2 upsertQuery syntax also 2021-03-30 10:51:52 +03:00
Chris Lu
a5662c9e5e adjust ordering 2021-03-30 00:26:57 -07:00
Chris Lu
bec74c3e6a
Merge pull request #1948 from LazyDBA247-Anyvision/master 
add enableUpsert=true
 2021-03-30 00:25:01 -07:00
LazyDBA247-Anyvision
9f1cab179c use a compatible pg syntax for upsert 2021-03-30 10:21:27 +03:00
Chris Lu
a95929e53c reduce default concurrentWriters to 32 2021-03-30 00:17:52 -07:00
LazyDBA247-Anyvision
7f44d953b5 fix GetBool 2021-03-30 01:36:02 +03:00
LazyDBA247-Anyvision
4c51e6a660 add enableUpsert=true 
and rename config to upsertQuery
 2021-03-30 00:32:03 +03:00
Chris Lu
b61e9f6a1a adjust ordering 2021-03-29 00:49:50 -07:00
Chris Lu
1ebdbad222
Merge pull request #1946 from LazyDBA247-Anyvision/master 
Adding Customisation - insertQuery for postgres/mysql
 2021-03-29 00:42:48 -07:00
Konstantin Lebedev
8a95f9c10c iam GetUser 2021-03-29 12:01:44 +05:00
LazyDBA247-Anyvision
96c62bd34d scaffold - insertQuery 
Example for upsert query for mysql & postgres variants
 2021-03-29 10:01:12 +03:00
LazyDBA247-Anyvision
4a02389eb0 Adding custom insertQuery support for postgres/2 mysql/2 2021-03-29 09:58:13 +03:00
Chris Lu
6b7aa9633f 2.36 2021-03-28 19:09:06 -07:00
Chris Lu
bd7471d877 refactor 2021-03-25 12:05:59 -07:00
李海
06be5dc6c3 log snowflake sequencer nodeid's hex when start, it'll be in part of new assigned fid 2021-03-25 18:59:40 +08:00
李海
69b2dab9c6 add a snowflake sequencer as more robust fid generator, but less compressable than small auto-inc id 2021-03-25 18:49:26 +08:00
Konstantin Lebedev
40938d6a47 SaveInsideFiler S3 Configuration 2021-03-25 12:34:11 +05:00
Konstantin Lebedev
9f26f2815c SaveAs S3 Configuration 2021-03-25 12:34:11 +05:00
Konstantin Lebedev
82b0463fac handler PutUserPolicy 
https://docs.aws.amazon.com/IAM/latest/APIReference/API_PutUserPolicy.html
 2021-03-25 12:34:11 +05:00
Konstantin Lebedev
d7719d0542 base handlers 2021-03-25 12:34:11 +05:00
Konstantin Lebedev
03c7953254 init Iam Api Server 2021-03-25 12:34:11 +05:00
Chris Lu
82dfe06066
Merge pull request #1933 from kmlebedev/missingKeysThreshold 
add missingKeysThreshold for fsck
 2021-03-24 10:56:25 -07:00
Konstantin Lebedev
020a5d40c3 avoid counter is null 2021-03-24 22:51:22 +05:00
Konstantin Lebedev
df6cf0a2fa nonRepairThreshold 2021-03-24 22:07:13 +05:00
Konstantin Lebedev
9ce82a9cd2 GOMAXPROCS use defaults to the value of runtime.NumCPU 
https://github.com/chrislusf/seaweedfs/issues/1942
 2021-03-24 18:45:58 +05:00
Konstantin Lebedev
60972f1c97 rename option repairThreshold 2021-03-24 13:24:49 +05:00
Chris Lu
a801332b0d filer: return 409 if file conflicts with a directory 
fix https://github.com/chrislusf/seaweedfs/issues/1938
 2021-03-24 01:03:11 -07:00
Chris Lu
c42b95c596 more help message 
fix https://github.com/chrislusf/seaweedfs/issues/1937
 2021-03-23 17:27:57 -07:00
Konstantin Lebedev
40a9e88c07 rename var 2021-03-23 21:42:35 +05:00
Konstantin Lebedev
d21a5bf5d0 add missingKeysThreshold for fsck 2021-03-23 15:04:07 +05:00
Chris Lu
85cff10787 mysql: ensure name column is case sensitive 
fix https://github.com/chrislusf/seaweedfs/issues/1931
 2021-03-23 00:46:50 -07:00
Chris Lu
4abb511db3 make a local copy of the in memory cached data 2021-03-22 22:33:07 -07:00
Chris Lu
3cbc40fa48 avoid creating multiple reader 2021-03-22 22:32:47 -07:00
Chris Lu
2f7c7afdec refactor: remove unused parameter 2021-03-22 22:13:19 -07:00
Chris Lu
d48dd0c738 use []byte directly instead of bytes.Buffer 2021-03-22 22:12:57 -07:00
Chris Lu
8eb14967ef shell: volume.check.disk adds optional slow mode 
fix https://github.com/chrislusf/seaweedfs/issues/1930
 2021-03-22 21:01:43 -07:00
Chris Lu
288369cfc7 mount: release resources when Forget() is called 
address https://github.com/chrislusf/seaweedfs/issues/1929
 2021-03-22 16:22:19 -07:00
Chris Lu
5d931eff27 avoid possible nil 
fix https://github.com/chrislusf/seaweedfs/issues/1928

The nil was because of `dn.Parent().UnlinkChildNode(dn.Id())` in topo.UnRegisterDataNode() function, when the dn leaves the cluster.
 2021-03-22 13:24:07 -07:00
Chris Lu
f315eb2bb8 fix build error 2021-03-22 00:12:53 -07:00
Chris Lu
1dd5bc134c 2.35 2021-03-22 00:05:09 -07:00
Chris Lu
b465095db1 shell: add volume.check.disk to fix inconsistency for replicated volumes 
fix https://github.com/chrislusf/seaweedfs/issues/1923
 2021-03-22 00:03:16 -07:00
Chris Lu
df461402cc ensure entry attribute is not nill 
fix https://github.com/chrislusf/seaweedfs/issues/1926
 2021-03-21 22:27:30 -07:00
Chris Lu
102a951377 refactor, split into 2 files 2021-03-21 13:05:53 -07:00
Chris Lu
f6e2566ee5 better logs for cookie mismatch 2021-03-20 12:02:57 -07:00
Chris Lu
27e24162f2 minor 2021-03-20 06:37:03 -07:00
Chris Lu
41cc6e661b fs.mv ensures there are 2 arguments 2021-03-20 06:34:13 -07:00
Chris Lu
b2c1c209a5 refactor 2021-03-20 06:34:13 -07:00
Chris Lu
e9b1853d63 go fmt 2021-03-19 01:31:56 -07:00
Chris Lu
b1a86cf808 s3: copy object to itself 
fix https://github.com/chrislusf/seaweedfs/issues/1922
 2021-03-19 01:31:49 -07:00
Konstantin Lebedev
bf94149920 add line 2021-03-17 23:41:34 +05:00
Konstantin Lebedev
c5705e7a35 dir + slash key does not exist 
https://github.com/chrislusf/seaweedfs/issues/1917
 2021-03-17 23:40:42 +05:00
Konstantin Lebedev
eb54993a4e Merge branch 'upstreamMaster' into check_chunkviews_mr 
# Conflicts:
#	weed/filer/filechunk_manifest.go
#	weed/filer/stream.go
#	weed/replication/repl_util/replication_util.go
#	weed/util/fasthttp_util.go
 2021-03-16 15:29:49 +05:00
Chris Lu
9672f9e1b2 2.34 2021-03-16 03:01:15 -07:00
Chris Lu
69694a17be reverting 7d57664c2d 2021-03-16 02:59:26 -07:00
Chris Lu
c00dd5e62e report error if entry not found 2021-03-16 02:55:49 -07:00
Konstantin Lebedev
06da02739d CheckAllChunkViews() for HEAD requests only 2021-03-16 14:15:17 +05:00
Chris Lu
91a3ac9731 2.33 2021-03-16 00:36:06 -07:00
Chris Lu
4b1ed227d1 revert fasthttp changes 
related to https://github.com/chrislusf/seaweedfs/issues/1907
 2021-03-16 00:33:14 -07:00
Chris Lu
10164d0386
Merge pull request #1905 from wuh-fnst/s3ListCounterErr 
s3: count correction for the number of files in the dir
 2021-03-15 20:54:58 -07:00
Chris Lu
22a07a12d2 [mount] [regression] filer.path option results in empty mount in 2.32 
fix https://github.com/chrislusf/seaweedfs/issues/1906
 2021-03-15 20:46:25 -07:00
wuh-fnst
050a7f9599 s3: count correction for the number of files in the dir 2021-03-16 09:22:03 +08:00
Konstantin Lebedev
3a3699867b Status PartialContent for Content-Range response 2021-03-15 23:30:22 +05:00
Konstantin Lebedev
90510e3137 If WriteHeader is not called explicitly, the first call to Write 
will trigger an implicit WriteHeader(http.StatusOK).
WriteHeader: Only one header may be written. Go does not currently!
 2021-03-15 21:22:59 +05:00
Konstantin Lebedev
46b9f5cff4 add debug logging 2021-03-15 19:33:33 +05:00
Konstantin Lebedev
7194a5e7bf avoid http error: superfluous response.WriteHeader 
https://github.com/chrislusf/seaweedfs/issues/1838
 2021-03-15 18:52:59 +05:00
Chris Lu
9f00f95bfb 2.32 2021-03-14 21:32:00 -07:00
Chris Lu
a4cfffc264 shell: fix moving volume, volume server evacuate 
fix https://github.com/chrislusf/seaweedfs/issues/1534
 2021-03-14 21:29:55 -07:00
Chris Lu
ca100568f7 fix parsing 2021-03-14 20:50:14 -07:00
Chris Lu
352ba23f83 revert previous change 
revert 29e62aba00
 2021-03-14 20:49:56 -07:00
Chris Lu
29e62aba00 possible fix for volume balance 
address https://github.com/chrislusf/seaweedfs/issues/1534
 2021-03-14 20:23:19 -07:00
Chris Lu
35d939a1d2 go fmt 2021-03-14 13:21:02 -07:00
Chris Lu
2d4c2db81d filer: leveldb, rocksdb auto create store directory 
fix https://github.com/chrislusf/seaweedfs/issues/1901
 2021-03-14 13:20:14 -07:00
Chris Lu
6d3a96eb56 filer: mysql2, postgres2 trigger actions on bucket creation and deletion 
fix https://github.com/chrislusf/seaweedfs/issues/1877
 2021-03-13 22:07:39 -08:00
Chris Lu
cb423312a4 prevent nil volume message 2021-03-13 11:05:29 -08:00
Chris Lu
972327f966 prevent nil volume nm 2021-03-13 11:04:51 -08:00
Chris Lu
2a68ddb963 default to empty host to bind to tcp4 and tcp6 2021-03-12 15:32:51 -08:00
Chris Lu
5fbcaaf37a fullpath() test is not needed 2021-03-12 09:45:25 -08:00
Chris Lu
7d57664c2d mount: internals switch to filer.Entry instead of protobuf 2021-03-12 00:36:38 -08:00
wuh-fnst
4c1d945e46 make List correctly judge whether it is the last file 2021-03-12 11:22:26 +08:00
Chris Lu
cca66c7fbe print out the url 2021-03-11 18:39:45 -08:00
Chris Lu
7f887c78d2 fix 2021-03-11 18:18:16 -08:00
Chris Lu
fcc52d1e4e shell: add s3.clean.uploads to clean up stale multipart uploads 
fix https://github.com/chrislusf/seaweedfs/issues/1855
 2021-03-11 18:05:24 -08:00
Chris Lu
40dc5ac904 mount: add a bit retry when connecting to filer during startup 2021-03-11 16:36:43 -08:00
Chris Lu
aba47fd9e8 reduce repeated calls to dir.FullPath() 2021-03-11 14:08:20 -08:00
Chris Lu
d084334ffd
Merge pull request #1883 from kmlebedev/passDelNonEmptyFolder 
s3 delete-objects pass OK for fail to delete non-empty folder
 2021-03-11 11:01:44 -08:00
Chris Lu
0e35836cc1
Merge pull request #1890 from kmlebedev/ZeroDataSize 
error if read chunk zero data size
 2021-03-11 10:51:48 -08:00
Konstantin Lebedev
58cdcc6d6e error if urls not found 2021-03-11 23:34:36 +05:00
Chris Lu
79280e1513 escape special characters 
fix https://github.com/chrislusf/seaweedfs/issues/1884
 2021-03-11 09:49:40 -08:00
Konstantin Lebedev
4db21012c1 error if read chunk zero data size 2021-03-11 22:38:59 +05:00
wuheng
828fbf3fb7 s3: "isLast" returns true when the file does not exist 2021-03-11 15:20:50 +08:00
Chris Lu
5d446673a6 fix error printing 2021-03-10 17:13:04 -08:00
Chris Lu
ca546f47e1 s3: escape object key if containing special characters 
fix https://github.com/chrislusf/seaweedfs/issues/1884
 2021-03-10 13:19:28 -08:00
Chris Lu
105fcd7bfc
Merge pull request #1872 from kmlebedev/grpc_auth 
TLS allowed CommonNames
 2021-03-10 11:17:49 -08:00
Konstantin Lebedev
bb3af2d70b no directoriesWithDeletion 2021-03-10 19:52:41 +05:00
Konstantin Lebedev
15b018da34 s3 delete-objects pass OK for fail to delete non-empty folder 2021-03-10 19:41:35 +05:00
Konstantin Lebedev
348e21a08c add comments 2021-03-10 14:42:39 +05:00
Konstantin Lebedev
831953c55c allowed wildcard domain 2021-03-10 14:02:13 +05:00
Konstantin Lebedev
4bf93d6e63 comma-separated 2021-03-10 12:43:13 +05:00
Konstantin Lebedev
0e02f7e258 comma-separated SSL certificate common names 2021-03-10 12:42:44 +05:00
Chris Lu
737bde5ab7 refactoring, remove duplicated code 2021-03-09 23:23:01 -08:00
Chris Lu
5ba4b479f8 properly lock file.entry object 
fix https://github.com/chrislusf/seaweedfs/issues/1882
 2021-03-09 23:08:38 -08:00
Chris Lu
5be83dd4a9 add comments 2021-03-09 14:13:48 -08:00
Chris Lu
9d8ca513d9 refactor 2021-03-09 13:21:26 -08:00
Chris Lu
f5b5e4da2d 2.31 2021-03-09 12:52:16 -08:00
Chris Lu
828f6e9f4d volume: auto add missing vif files 
fix https://github.com/chrislusf/seaweedfs/issues/1878
 2021-03-09 12:09:32 -08:00
Chris Lu
387c6f4218 Revert "delete the folder object in multi_object_delete" 
This reverts commit 2e89c8c9ae.
 2021-03-09 10:07:27 -08:00
qieqieplus
5b16820924 fix: collection stats won't update if all volumes expired at same iteration 2021-03-09 18:49:45 +08:00
Konstantin Lebedev
190fada1ef TLS allowed commonNames 2021-03-08 21:39:44 +05:00
Konstantin Lebedev
c6d3735605 permitCommonNames 
https://github.com/chrislusf/seaweedfs/issues/1841
https://jbrandhorst.com/post/grpc-auth/
 2021-03-08 13:16:17 +05:00
Chris Lu
d888e9a90f 2.30 2021-03-07 15:01:19 -08:00
Chris Lu
60686a0bb7 make tcp optional 2021-03-07 14:45:36 -08:00
Chris Lu
3b2a9c98ef fix nil 2021-03-07 11:45:35 -08:00
Chris Lu
df72dc206d stats collect chunk upload retry count 2021-03-07 11:26:15 -08:00
Chris Lu
726edab054 avoid nil when closing an index 
fix https://github.com/chrislusf/seaweedfs/issues/1870
 2021-03-07 11:03:09 -08:00
Chris Lu
bdfed16d42 avoid nil exception 
fix https://github.com/chrislusf/seaweedfs/issues/1869
 2021-03-07 01:49:06 -08:00
Chris Lu
1bd880dcdb adds tcp writes benchmark 2021-03-06 14:26:27 -08:00
Chris Lu
9d402ebe9f refactoring 2021-03-06 14:26:27 -08:00
Chris Lu
e9eddfb8a2 remove wrong writes 2021-03-06 14:26:27 -08:00
Chris Lu
8363be8548 trim out trailing return character 2021-03-06 14:26:27 -08:00
Chris Lu
38fc200e56 CRCWriter consistent with CRC 2021-03-06 14:26:27 -08:00
Chris Lu
1444e9d275 migrated multi host connection pool from godropbox package 
removing unneeded dependencies, which involved etcd versions.
 2021-03-06 14:26:27 -08:00
Patrick Schmidt
7413d59750 Fix EC shard count logic 
This fixes the calculation of the amount of EC shards a node holds.
Previously a global counter was increased, but also used inside the
loop to apply disk usage deltas. This led to wrong absolute numbers.
The fix is to apply only deltas of single EC shards per iteration.
 2021-03-05 12:50:58 +01:00
Chris Lu
400de380f4 volume server: support tcp direct put/get/delete 2021-03-05 02:29:38 -08:00
Chris Lu
2e89c8c9ae delete the folder object in multi_object_delete 
fix https://github.com/chrislusf/seaweedfs/issues/1857
 2021-03-04 14:03:40 -08:00
Chris Lu
c5cb346a08 fix compilation error 2021-03-04 12:36:20 -08:00
Chris Lu
f5cc96f289 skip checking master peers if not starting master 2021-03-04 12:31:26 -08:00
Chris Lu
40ff30b83f adjust help message 2021-03-03 02:12:46 -08:00
Chris Lu
c0842fe99f add filer.meta.backup command 2021-03-03 02:02:29 -08:00
Chris Lu
0c5c51eb98 adjust help message 2021-03-02 23:07:39 -08:00
Chris Lu
bd727b7b53 go fmt 2021-03-02 20:59:56 -08:00
Chris Lu
e982b06bcd support IPv6 2021-03-02 20:59:39 -08:00
Chris Lu
bcf32591b7 reduce possibility of nil entry 2021-03-02 13:33:56 -08:00
Chris Lu
be9c7c21ec reduce possibility of nil file entry 2021-03-02 11:17:17 -08:00
Chris Lu
02e146f3e1 return err 2021-03-02 08:54:18 -08:00
Chris Lu
5511722420 s3: list permission is needed to list my buckets 
fix https://github.com/chrislusf/seaweedfs/issues/1837
 2021-03-01 12:41:55 -08:00
Chris Lu
c46d16b2f3 skip network error when remote volumes start 2021-03-01 01:20:06 -08:00
Chris Lu
6e43e8ce17 add logs for remote file loading 2021-03-01 00:48:59 -08:00
Chris Lu
015d16f43f add vif file versions in case loading superblock fails 2021-03-01 00:48:30 -08:00
Chris Lu
d680676d45 skip already loaded backends 2021-03-01 00:47:03 -08:00
Chris Lu
540441fd38 go fmt 2021-02-28 20:34:14 -08:00
Chris Lu
c9722dceb2 adjust log level 2021-02-28 20:26:55 -08:00
Chris Lu
1b1c018165 adjust text 2021-02-28 19:02:43 -08:00
Chris Lu
2cca07b44c cloud tier to non-AWS s3 gateways 2021-02-28 18:59:09 -08:00
Chris Lu
f29ae0db76 2.29 2021-02-28 18:08:26 -08:00
bingoohuang
5cdff56731 set default env prefix to WEED_ 2021-03-01 09:35:23 +08:00
Chris Lu
e52c94640e filer.backup: added to replace filer.replicate 2021-02-28 16:22:27 -08:00
Chris Lu
014a31d11a minor 2021-02-28 16:21:09 -08:00
Chris Lu
f2fcb77808 local file sink: create backup files with permission 0755 2021-02-28 16:20:47 -08:00
Chris Lu
984fdd6192 always use non bucket prefixing url 2021-02-28 16:20:13 -08:00
Chris Lu
9abb041763 filer source: support filerProxy mode 2021-02-28 16:19:47 -08:00
Chris Lu
678c54d705 data sink: add incremental mode 2021-02-28 16:19:03 -08:00
Chris Lu
4ff2c5c4c9 rename file 2021-02-28 16:14:21 -08:00
Chris Lu
3b76a51f5f add help message 2021-02-27 12:15:49 -08:00
Chris Lu
70434df105 use "options" 2021-02-27 12:12:53 -08:00
Chris Lu
d3c31c69a7 avoid confusion: conf and config are too similar 2021-02-27 12:08:09 -08:00
Chris Lu
543c8daa6d
Merge pull request #1844 from kmlebedev/processRangeRequest 2021-02-26 09:18:22 -08:00
Patrick Schmidt
5f7b024891 Show the real disk usage in stats calls 
Currently the file size of only one volume location is taken into
account in the stats. This commit multiplies the disk usages by the
amount of nodes holding a replica of the volume.
This will yield the expected amount of disk usage and matches the
total size calculations from before.
 2021-02-26 13:58:40 +01:00
Konstantin Lebedev
ee21c0042e log error write entry 2021-02-26 14:18:01 +05:00
Chris Lu
7ba75e3d5a filer: do not return no content for empty files 
fix https://github.com/chrislusf/seaweedfs/issues/1831
fix https://github.com/chrislusf/seaweedfs/issues/1830
 2021-02-23 12:25:27 -08:00
Chris Lu
9edd964627 volume.tier.move: avoid repeated move for replicated volumes 
fix https://github.com/chrislusf/seaweedfs/issues/1792#issuecomment-784139348
 2021-02-23 03:49:14 -08:00
Chris Lu
37f104f88f 2.28 2021-02-22 22:54:34 -08:00
Chris Lu
90cdf9dcac avoid conflict with "weed scaffold -config=xxx" 2021-02-22 16:57:27 -08:00
Chris Lu
72b0d9d8c4 avoid unnecessary user home checking 2021-02-22 16:50:56 -08:00
Chris Lu
2270737344 volume: avoid fixed vacuum timeout for large volumes 
1GB for 3 minutes, about 5.7MB/s
 2021-02-22 12:52:37 -08:00
Chris Lu
44bdfb2d15 filer: avoid encryption and compression at the same time 
fix https://github.com/chrislusf/seaweedfs/issues/1828
 2021-02-22 12:22:49 -08:00
Chris Lu
62191b08ea disk type support custom tags 2021-02-22 02:03:12 -08:00
Chris Lu
30b30b8fe0 volume.tier.move: passing non-empty disk type 2021-02-22 01:59:03 -08:00
Chris Lu
5da63e045e avoid moving to another server with the same volume id 2021-02-22 01:44:18 -08:00
Chris Lu
6a4546d2c0 shell: add volume.tier.move 2021-02-22 01:30:07 -08:00
Chris Lu
1c233ad986 refactoring 2021-02-22 00:28:42 -08:00
Chris Lu
151c281f36 2.27 2021-02-21 19:29:27 -08:00
Chris Lu
258e93bc86
Merge pull request #1823 from bingoohuang/master 
feature: support command line arguments in a configuration file
 2021-02-20 22:35:18 -08:00
Chris Lu
03c643aa6b fix test 
fileSize need to be divided by 8
 2021-02-20 21:19:21 -08:00
Chris Lu
7635f6b9fa disk file avoid file.Stat() 2021-02-20 20:06:06 -08:00
bingoo
1af6c96cd0 fix log help 2021-02-21 10:46:01 +08:00
Chris Lu
a2383b3b12 Revert "Revert "avoid file.Stat()"" 
This reverts commit 7ef4c24f18.
 2021-02-20 12:42:09 -08:00
Chris Lu
7ef4c24f18 Revert "avoid file.Stat()" 
This reverts commit 98c93ca465.
 2021-02-20 12:39:33 -08:00
Chris Lu
1ad3200094 skip seek() when index file writes 2021-02-20 12:39:25 -08:00
Chris Lu
98c93ca465 avoid file.Stat() 
avoid one Syscall, but did not help on performance though
 2021-02-20 09:44:17 -08:00
bingoohuang
ee082ae402 revert comment temporarily 2021-02-20 16:59:38 +08:00
bingoohuang
ee7cdf3668 revert ErrXyz to ErrorXyz temporarily. 2021-02-20 16:58:08 +08:00
bingoohuang
50df484d86 support command line arguments in a configuration file so we can weed master -config=master.conf, weed volume -config=volume.conf and etc. 2021-02-20 16:52:57 +08:00
bingoohuang
f69356f589
Merge pull request #3 from chrislusf/master 2021-02-20 16:45:02 +08:00
Chris Lu
565f7a6e72 Update data_node.go 2021-02-19 14:22:36 -08:00
Chris Lu
a37473ae60 add back volume ids 
address https://github.com/chrislusf/seaweedfs/issues/1792#issuecomment-782339576
 2021-02-19 14:22:12 -08:00
Chris Lu
a1210d1e8d Revert "volume: avoid file.stat(), file.seek() if possible during writes" 
This reverts commit c78409a598.
 2021-02-19 03:56:27 -08:00
Chris Lu
b961cd6208 add WIP message 2021-02-19 03:39:19 -08:00
Chris Lu
f465d63b5d grpc should fail when heart beating to master 
fix https://github.com/chrislusf/seaweedfs/issues/1820
 2021-02-19 03:37:29 -08:00
Chris Lu
c576ad04ac fix volume server display for volumes 2021-02-19 01:38:56 -08:00
bingoohuang
7ffe736d20 renaming to ErrXyz 2021-02-19 15:59:55 +08:00
Chris Lu
c78409a598 volume: avoid file.stat(), file.seek() if possible during writes 2021-02-18 23:22:54 -08:00
Chris Lu
29575dadc5 remove extra files 2021-02-18 19:14:58 -08:00
Chris Lu
73958e357d add descriptive error if no free volumes 2021-02-18 19:10:20 -08:00
Chris Lu
776f497469 filer: fs.configure should try to read from entry.content also 
related to https://github.com/chrislusf/seaweedfs/issues/1792
 2021-02-18 17:07:02 -08:00
Chris Lu
e1992c83de webdav adds disktype 2021-02-18 15:45:44 -08:00
Chris Lu
c2ad6f1047 webdav add replication setting 
fix https://github.com/chrislusf/seaweedfs/issues/1817
 2021-02-18 12:15:09 -08:00
Chris Lu
816ef0ed30 add more help message 
fix https://github.com/chrislusf/seaweedfs/issues/1816
 2021-02-18 10:34:49 -08:00
bingoohuang
352ac2f271 Merge remote-tracking branch 'origin/master' 2021-02-18 14:05:51 +08:00
bingoohuang
eab6e31d34 use backticks instead of double quotes to avoid escaped additionally in regex 2021-02-18 14:05:28 +08:00
Chris Lu
3575d41009 go fmt 2021-02-17 20:57:08 -08:00
Chris Lu
6daa932f5c refactoring to get master function, instead of passing master values directly 
this will enable retrying later
 2021-02-17 20:55:55 -08:00
Chris Lu
dd9f3a0104 add sleep between upload retries 2021-02-17 20:55:11 -08:00
Chris Lu
aa17311063 s3: set filename when uploading file chunks 2021-02-17 20:54:53 -08:00
Chris Lu
3f8b0da677 filer: do not print password on error 
fix https://github.com/chrislusf/seaweedfs/issues/1809
 2021-02-17 02:13:52 -08:00
Chris Lu
cd866664a8 skip JWT if fileId is empty 
related to https://github.com/chrislusf/seaweedfs/issues/1808
 2021-02-16 15:39:12 -08:00
Chris Lu
632c94d438 Merge branch 'extend_to_disk_type' 2021-02-16 11:06:13 -08:00
Chris Lu
0611233f16 adjust printing 2021-02-16 10:55:30 -08:00
Chris Lu
68775d29e3 fix tests 2021-02-16 10:51:03 -08:00
Chris Lu
7403cd43c5 only use "" for hdd to avoid two values for the same thing 2021-02-16 10:50:42 -08:00
Chris Lu
b314d78e97 fix print 2021-02-16 10:48:28 -08:00
Chris Lu
53ca7e66ef avoid dead lock 2021-02-16 10:48:16 -08:00
Chris Lu
38efc6f572 simplify 2021-02-16 10:47:32 -08:00
Chris Lu
ad70107eb4 ec balance among nodes with HDD 2021-02-16 06:22:50 -08:00
Chris Lu
3097b9a9b7 fix existence checking 2021-02-16 05:59:43 -08:00
Chris Lu
ebf320ec22 avoid divided by zero 2021-02-16 05:59:24 -08:00
Chris Lu
a0c6db361c avoid nil 2021-02-16 05:33:38 -08:00
Chris Lu
36f95e50a9 avoid possible nil disk info 2021-02-16 05:13:48 -08:00
Chris Lu
43101ccea0 move to the empty nodes first 2021-02-16 04:27:16 -08:00
Chris Lu
38bbef7ec1 avoid nil 2021-02-16 04:16:46 -08:00
Chris Lu
cb9cc29518 volume.list display; fix updating maxVolumeCount for disk 2021-02-16 03:55:24 -08:00
Chris Lu
3fe628f04e use hdd instead of empty string 2021-02-16 03:03:00 -08:00
Chris Lu
f8446b42ab this can compile now!!! 2021-02-16 02:47:02 -08:00
Chris Lu
b9b5b932c5 filer: postgres2 mysql2 avoid repeatedly creating tables 2021-02-15 16:19:24 -08:00
Chris Lu
71f0c19515 2.26 2021-02-15 13:38:38 -08:00
Chris Lu
8b169aa47c s3: Critical - S3 Delete Multiple Objects, delete the bucket 
fix https://github.com/chrislusf/seaweedfs/issues/1806
 2021-02-15 13:38:29 -08:00
Chris Lu
50202d1391
Merge pull request #1805 from LazyDBA247-Anyvision/master 
better postgresql connection pool management
 2021-02-14 22:26:29 -08:00
Chris Lu
8726ef34c1 2.25 2021-02-14 22:23:43 -08:00
LazyDBA247-Anyvision
7f458d5e78 better postgres connection pool management 
adding SetConnMaxLifetime configuration (https://golang.org/pkg/database/sql/#DB.SetConnMaxLifetime)
to enable refresh of stale connections.
 2021-02-15 07:45:09 +02:00
Chris Lu
0bc3a1f9e8 disk type only supports hdd and ssd, not ready for random tags yet 2021-02-14 11:38:43 -08:00
LazyDBA247-Anyvision
51b4963e2e postgres2 & memsql2 
add escape (quote identifiers) for the dynamic sql
so tables (collections) with special characters will work.
 2021-02-14 13:14:36 +02:00
Chris Lu
876dbe3d26 collect disk types 2021-02-13 23:25:16 -08:00
Chris Lu
861f8fab9b refactoring 2021-02-13 22:34:12 -08:00
Chris Lu
881f05f5a0 adjust help message 2021-02-13 16:13:52 -08:00
Chris Lu
ef76365ec2 adjust help message 2021-02-13 15:47:08 -08:00
Chris Lu
7ce647f27e support customizable disk type 2021-02-13 15:42:42 -08:00
Chris Lu
4bd8a692d8 disk type can be generic tags 2021-02-13 13:50:14 -08:00
Chris Lu
712b3e9e53 mount: release need to avoid nil reader 
fix https://github.com/chrislusf/seaweedfs/issues/1803
 2021-02-13 13:39:37 -08:00
Chris Lu
4ce56bac08 fix format error 2021-02-12 06:44:26 -08:00
Chris Lu
0604afcda7 mount: remove unnecessary flush on release 2021-02-12 06:01:04 -08:00
Chris Lu
7d9dc3c6a2 use fasthttp lib to read 2021-02-12 05:59:52 -08:00
Chris Lu
487e435679 adjust http max idle connections per host 
related to https://github.com/chrislusf/seaweedfs/issues/1802
 2021-02-12 03:47:15 -08:00
Chris Lu
0f426ce34d
Merge pull request #1801 from kmlebedev/recoveringRabbitMQ 
Do reconnect to RabbitMQ
 2021-02-11 01:05:03 -08:00
Konstantin Lebedev
94eac4f00e Do reconnect to RabbitMQ 
https://github.com/google/go-cloud/issues/2958
https://github.com/chrislusf/seaweedfs/issues/1773
 2021-02-11 13:59:36 +05:00
Chris Lu
885ca34748 volume: fail fast if idx files are missing 
fix https://github.com/chrislusf/seaweedfs/issues/1796
 2021-02-11 00:44:40 -08:00
Chris Lu
a0e84c4fbc go fmt 2021-02-10 23:41:05 -08:00
Chris Lu
c959aaa19b mount: load directory based on correct filer path 
fix https://github.com/chrislusf/seaweedfs/issues/1799
 2021-02-10 23:14:18 -08:00
Konstantin Lebedev
0cfed8c3cb restart replication if RabbitMQ connection closed 
https://github.com/google/go-cloud/issues/2958
 2021-02-10 15:29:05 +05:00
Chris Lu
770393a48c volume: add capability to change disk type when moving a volume 2021-02-09 23:58:08 -08:00
Chris Lu
821c46edf1 Merge branch 'master' into support_ssd_volume 2021-02-09 11:37:07 -08:00
Chris Lu
a833021132 fix refactoring left over 2021-02-07 23:03:03 -08:00
Chris Lu
5e4b5109dd 2.24 2021-02-07 22:29:43 -08:00
Chris Lu
15c60cbb26 close the grpc connection after 10 hours 
related to https://github.com/chrislusf/seaweedfs/issues/1782
 2021-02-07 03:50:01 -08:00
bingoohuang
7256902fb0 fix typo offset.ToAcutalOffset to offset.ToActualOffset 2021-02-07 12:11:51 +08:00
bingoohuang
94ea3bd3a5 renaming NeedleMapType to NeedleMapKind 2021-02-07 09:00:03 +08:00
Chris Lu
c3af72d950 fier store: fix elastic search regression 
fix https://github.com/chrislusf/seaweedfs/issues/1774
 2021-02-04 01:30:14 -08:00
Konstantin Lebedev
2fb4c60540 fix log warn 2021-02-03 21:34:32 +05:00
Konstantin Lebedev
24ec17219b RabbitMQ delay retry with Dead Letter Exchange 
https://github.com/chrislusf/seaweedfs/issues/1773

https://github.com/google/go-cloud/issues/2952
 2021-02-03 21:32:30 +05:00
Chris Lu
a331bbb3ae filer: should return 204 on DELETE to nonexistent file 
related to

https://github.com/chrislusf/seaweedfs/issues/1776
https://github.com/chrislusf/seaweedfs/issues/1160
 2021-02-03 00:40:31 -08:00
Chris Lu
5353e38469 S3: should return 204 on DELETE to nonexistent file 
fix https://github.com/chrislusf/seaweedfs/issues/1776
 2021-02-03 00:35:44 -08:00
Chris Lu
2396ac234c filer: fix elastic search pagination 
possible fix for https://github.com/chrislusf/seaweedfs/issues/1774
 2021-02-02 11:34:12 -08:00
Chris Lu
609daaf387 s3: DeleteMultipleObjectsHandler clean up leftover empty folders 
fix https://github.com/chrislusf/seaweedfs/issues/1772
 2021-02-01 10:49:17 -08:00
Chris Lu
d67ccb66c0 webdav: can start together with "weed server" or "weed filer" 2021-01-31 22:16:52 -08:00
Chris Lu
76430790b9 webdav: cache to version specific folder 2021-01-31 22:02:03 -08:00
Chris Lu
318a3d2efc 2.23 2021-01-31 20:18:06 -08:00
Chris Lu
4be51c0701 filer: leveldb and hbase may miss files when listing large directories more than 1024 
fix https://github.com/chrislusf/seaweedfs/issues/1768
 2021-01-31 20:11:44 -08:00
Chris Lu
1102ae32c4 fix concurrent map reads 2021-01-31 18:26:26 -08:00
Chris Lu
314dc1c957 filer: etcd store fix listing 
fix https://github.com/chrislusf/seaweedfs/issues/1767
 2021-01-30 13:38:44 -08:00
Chris Lu
d475c89fcc go fmt 2021-01-28 15:23:46 -08:00
Chris Lu
19295600f9 mount: change option name to volumeServerAccess, with publicUrl and filerProxy modes 2021-01-28 15:23:16 -08:00
Chris Lu
990fa69bfe add back AdjustedUrl() related code 2021-01-28 14:36:29 -08:00
Chris Lu
9292796ec2 fmt 2021-01-28 14:29:14 -08:00
Chris Lu
c2bf1a88ac delete from the deepest directory first when checking empty folders 2021-01-28 14:28:40 -08:00
Chris Lu
e9d8201925 s3: batch purge empty folders 2021-01-28 13:20:06 -08:00
Chris Lu
cf252fc0cd mount: report error when Flush() 
The error in Release() is not working.

See https://github.com/jaderhs/libfuse/blob/master/FAQ

related to https://github.com/chrislusf/seaweedfs/issues/1765
 2021-01-28 04:46:37 -08:00
Chris Lu
9a06c35da4 replicate: incremental sink only contains new and updated files 
address da08402ba2
 2021-01-28 02:39:22 -08:00
Chris Lu
da08402ba2 replicate: use creation time for local incremental file sink 
related to https://github.com/chrislusf/seaweedfs/pull/1762
 2021-01-28 02:17:41 -08:00
Chris Lu
822f1ade9d
Merge pull request #1762 from kmlebedev/backupsink 
replication to create time date directory
 2021-01-28 02:03:33 -08:00
Konstantin Lebedev
be1062b7fc rename in scaffold 2021-01-28 14:59:20 +05:00
Konstantin Lebedev
02fdc0a333 rename backup to local_incremental and use mtime 2021-01-28 14:56:13 +05:00
Chris Lu
5f72c388bc
Merge pull request #1764 from danlsgiga/handle-umask 
Use provided umask on mount
 2021-01-28 01:09:20 -08:00
Daniel Santos
d38613c49a match upstream 2021-01-27 20:56:41 -07:00
Chris Lu
b81956bcb5 mount: invalidate kernel cache when mounted to a filer path 
fix https://github.com/chrislusf/seaweedfs/issues/1752#issuecomment-768178422
 2021-01-27 10:28:37 -08:00
Konstantin Lebedev
6b54ff9912 replication to create time date directory 2021-01-27 15:01:33 +05:00
Chris Lu
f20ec82a28 update help message 
see https://github.com/chrislusf/seaweedfs/issues/1758#issuecomment-768123224
 2021-01-27 00:48:31 -08:00
Chris Lu
1045ab8b07 avoid error message 2021-01-27 00:21:59 -08:00
Chris Lu
0488c84bea adjust help message 2021-01-27 00:17:52 -08:00
Konstantin Lebedev
3634811408 replication to todays date directory 2021-01-27 12:45:58 +05:00
Chris Lu
5e07afb0f0
Merge pull request #1759 from kmlebedev/sink.local 
replication to local disk storage
 2021-01-26 22:31:54 -08:00
Chris Lu
b3f66199db shell: volume.fix.replication adds collectionPattern with wildcard characters 
fix https://github.com/chrislusf/seaweedfs/issues/1758
 2021-01-26 22:30:39 -08:00
Daniel Santos
0c64d60fdb Fix log message with correct mode 2021-01-26 21:45:52 -07:00
Daniel Santos
0aaaa9b470 Fix var assignment 2021-01-26 21:06:08 -07:00
Daniel Santos
5a761bb7a4 Enforce umask correctly 2021-01-26 20:42:10 -07:00
Chris Lu
ad2a20c8a5 notification add ack and nack 2021-01-26 11:08:44 -08:00
Konstantin Lebedev
612b7975a1 replication to local disk storage 2021-01-26 22:50:25 +05:00
Chris Lu
3a1d3d3413 mount: properly invalidate kernel node cache entry 
fix https://github.com/chrislusf/seaweedfs/issues/1752
 2021-01-26 02:50:53 -08:00
Chris Lu
cc1f3907ff filer.sync: default filerProxy to true 2021-01-25 10:14:42 -08:00
Chris Lu
62ebb917e0 mount: avoid the need to specify different cache directory for different mount 
fix https://github.com/seaweedfs/seaweedfs-csi-driver/issues/11
 2021-01-25 09:10:20 -08:00
Chris Lu
17f7c1c43f Merge branch 'mount_from_outside_cluster' 2021-01-24 19:03:02 -08:00
Chris Lu
00707ec00f mount: outsideContainerClusterMode proxy through filer 
Running mount outside of the cluster would not need to expose all the volume servers to outside of the cluster. The chunk read and write will go through the filer.
 2021-01-24 19:01:58 -08:00
Chris Lu
9bcb28a3ea avoid possible nil reader 
fix https://github.com/chrislusf/seaweedfs/issues/1754
 2021-01-24 17:19:35 -08:00
Chris Lu
2c5eac5705 2.22 2021-01-24 14:06:09 -08:00
Chris Lu
6ca10725b8 Revert "mount: when outside cluster network, use filer as proxy to access volume servers" 
This reverts commit 096e088d7b.
 2021-01-24 03:15:19 -08:00
Chris Lu
096e088d7b mount: when outside cluster network, use filer as proxy to access volume servers 2021-01-24 01:41:38 -08:00
Chris Lu
ccbdb38c89 s3: fix for listing objects if more than 1000 in the folder 
many existing tools have max key set to 1000
 2021-01-24 00:27:52 -08:00
Chris Lu
80b8692688 filer.sync: replicate outside of either cluster, only need to see filers 2021-01-24 00:01:44 -08:00
Chris Lu
f8dbb03bdd filer: append operation returns final file size 2021-01-23 04:19:49 -08:00
Chris Lu
759482e2ac filer: proxy all http methods to volume servers 2021-01-23 03:52:38 -08:00
Chris Lu
937cfacc01 filer: add "proxyToFileId" to reverse proxy to a volume server 2021-01-23 03:43:48 -08:00
Chris Lu
20ef3bb8d4 mount: use direct_io to avoid OS page cache 
fix https://github.com/chrislusf/seaweedfs/issues/1752
 2021-01-22 22:39:46 -08:00
Chris Lu
711c3f3939 filer: fix http status 206 setting 
fix https://github.com/chrislusf/seaweedfs/issues/1753
 2021-01-22 00:26:30 -08:00
Chris Lu
024c186a48 filer.meta.tail: output json format for easier parsing 2021-01-21 22:39:25 -08:00
Chris Lu
84f05787f8 mount: limit background requests and congestion threshold 2021-01-20 23:30:14 -08:00
Chris Lu
f98817cfe6 filer: support appending to a file 2021-01-20 13:40:32 -08:00
Chris Lu
1b8e3da295 refactoring 2021-01-20 12:59:54 -08:00
Chris Lu
1a3f859c58 filer store: postgres configuration adjust text 2021-01-19 18:12:07 -08:00
Chris Lu
b434f7e4e0 filer store: postgres adjust default config, add optioanl schema 2021-01-19 18:10:36 -08:00
Chris Lu
d5add83e85 filer store: add postgres2 2021-01-19 18:07:29 -08:00
Chris Lu
52a8f1470e filer store: add mysql2 2021-01-19 17:21:50 -08:00
Chris Lu
93b3adba98 fix bucket creation 2021-01-19 15:55:51 -08:00
Chris Lu
fa0c8d5283 fix error 2021-01-19 14:05:48 -08:00
Chris Lu
4c5b752b04 restructuring sql stores 2021-01-19 13:53:16 -08:00
Chris Lu
96354208c5 scaffold: add YugabyteDB 2021-01-19 12:35:14 -08:00
Chris Lu
ca8f793978 refactor: prepare for bucket specific tables 2021-01-19 12:34:58 -08:00
Chris Lu
90ce1eec19 add more input params 2021-01-19 11:32:55 -08:00
Chris Lu
f13c082583 add memsql to comments 2021-01-19 11:31:55 -08:00
Chris Lu
e439b65e38 fix test 2021-01-18 01:48:00 -08:00
Chris Lu
c7197470fd 2.21 2021-01-18 01:25:04 -08:00
Chris Lu
5a3386e39b Revert "Revert "mount: fake support for socket/block/character/fifo devices"" 
This reverts commit 3d5cb7eb86.
 2021-01-18 01:15:07 -08:00
Chris Lu
67faa56920 Revert "Revert "mount: add more entry type"" 
This reverts commit 4085b79d38.
 2021-01-18 01:14:58 -08:00
Chris Lu
7119e7b491 mount: ensure reading from dirty pages 2021-01-18 01:14:42 -08:00
Chris Lu
de876c795d minor fix 2021-01-18 01:14:27 -08:00
Chris Lu
4085b79d38 Revert "mount: add more entry type" 
This reverts commit d7d907be27.
 2021-01-18 00:19:19 -08:00
Chris Lu
3d5cb7eb86 Revert "mount: fake support for socket/block/character/fifo devices" 
This reverts commit 61ef2d8658.
 2021-01-18 00:18:57 -08:00
Chris Lu
61ef2d8658 mount: fake support for socket/block/character/fifo devices 
to pass pjdfstest. A distributed file system can not really support these things anyway.
 2021-01-17 23:51:10 -08:00
Chris Lu
d7d907be27 mount: add more entry type 2021-01-17 23:46:39 -08:00
Chris Lu
389426bbb7 s3: listing may repeat on the edge 2021-01-17 22:59:31 -08:00
Chris Lu
09f49d1c04 refactoring 2021-01-16 19:52:15 -08:00
Chris Lu
a4063a5437 add stream list directory entries 2021-01-15 23:56:24 -08:00
Chris Lu
01dc8a43ba fix rocksdb 2021-01-15 18:31:29 -08:00
Chris Lu
16ad74f477 go fmt 2021-01-14 23:11:27 -08:00
Chris Lu
f002e668de change limit to int64 in case of overflow 2021-01-14 23:10:37 -08:00
Chris Lu
19e52fd414 pass along hasMore 2021-01-14 22:49:29 -08:00
Chris Lu
c0bcf6f6e1 implement sql changes 2021-01-14 22:44:22 -08:00
Chris Lu
5d4568b91f implement elastic changes 2021-01-14 22:42:25 -08:00
Chris Lu
893cbc8482 implement c* changes 2021-01-14 22:38:34 -08:00
Chris Lu
5ef43b9b09 implement etcd changes 2021-01-14 22:35:56 -08:00
Chris Lu
54527f0326 implement hbase changes 2021-01-14 22:34:35 -08:00
Chris Lu
b5ceffe188 implement leveldb changes 2021-01-14 22:33:05 -08:00
Chris Lu
c64bfb0e2e implement mongodb changes 2021-01-14 22:28:51 -08:00
Chris Lu
34a846009d implement rocksdb changes 2021-01-14 22:24:26 -08:00
Chris Lu
d2ece29fff implement redis changes 2021-01-14 22:21:31 -08:00
Chris Lu
9a50dbcda0 chagned api 2021-01-14 22:18:38 -08:00
Chris Lu
e1c7bc66f9 change interface 2021-01-14 22:17:49 -08:00
Chris Lu
4aaa97c789 skip not found error in case of race condition 2021-01-14 19:56:14 -08:00
Chris Lu
d0826c2f62 mount: avoid showing "topics" folder 
fix https://github.com/chrislusf/seaweedfs/issues/1325
 2021-01-14 19:03:15 -08:00
Chris Lu
698f58f7c4 filer, s3: add http status 206 as late as possible 
fix https://github.com/chrislusf/seaweedfs/issues/1746
 2021-01-14 02:59:40 -08:00
Chris Lu
fd1d8a2a09 fix compilation 2021-01-14 00:06:15 -08:00
Chris Lu
ae812c07d6
Merge pull request #1743 from LazyDBA247-Anyvision/master 
MYSQL - Better Connection Pool Control
 2021-01-14 00:04:44 -08:00
Chris Lu
f20743bb38 remove WIP status 2021-01-13 22:21:34 -08:00
LazyDBA247-Anyvision
8eed763b97 better mysql connection pool management 
adding SetConnMaxLifetime configuration (https://golang.org/pkg/database/sql/#DB.SetConnMaxLifetime)
to enable refresh of connections.
 2021-01-14 08:14:21 +02:00
Chris Lu
f17aa1d06c skip directory loop if dropping the bucket 2021-01-13 13:49:04 -08:00
Chris Lu
ca73013453 leveldb3: ensure repeatable adding removing buckets 2021-01-13 13:48:48 -08:00
Chris Lu
e2c7e3fe6d better locks 2021-01-13 13:20:33 -08:00
Chris Lu
819a85f59a avoid viper concurrent access 2021-01-13 13:14:52 -08:00
Chris Lu
3fb2ed9093 filer.meta.tail: optionally submit metadata changes to ElasticSearch 2021-01-13 03:31:37 -08:00
Chris Lu
c5df2577f5 rename 2021-01-13 00:31:19 -08:00
Chris Lu
90c5077610 rename "weed watch" to "weed filer.meta.tail" 2021-01-12 18:48:01 -08:00
Chris Lu
5cd9c9e642 remove println 2021-01-12 18:47:38 -08:00
Chris Lu
def6d8e565 fix test 2021-01-12 12:32:54 -08:00
Chris Lu
1c7faf1100 Revert "close opened file" 
This reverts commit 2880160113.
 2021-01-12 11:07:14 -08:00
Chris Lu
2880160113 close opened file 2021-01-12 11:03:08 -08:00
Chris Lu
0ef43a23a7 go fmt 2021-01-12 02:30:12 -08:00
Chris Lu
629c9962e7 filer: add leveldb3 
support dedicated leveldb instance for each bucket
 2021-01-12 02:29:44 -08:00
Chris Lu
361043e6c1 filer store: leveldb2 fix nil entry error if not found 2021-01-12 02:28:57 -08:00
Chris Lu
cfb9342a15 avoid concurrent map updates to viper 2021-01-12 02:28:13 -08:00
Chris Lu
38d516251e Revert "mount: avoid fatal error" 
This reverts commit a4f0bac9b7.
 2021-01-11 13:50:15 -08:00
Chris Lu
a4f0bac9b7 mount: avoid fatal error 2021-01-11 13:26:59 -08:00
Chris Lu
8a7302ce37 s3: KeyCount is zero 
fix https://github.com/chrislusf/seaweedfs/issues/1741
 2021-01-11 03:50:45 -08:00
Chris Lu
2c7148cd55 redis cluster: do not enable route by latency by default 2021-01-11 02:43:53 -08:00
Chris Lu
acc0d5cd2c Revert "Revert "filer store: redis module upgrade"" 
This reverts commit dd0171e35c.
 2021-01-11 02:30:19 -08:00
Chris Lu
dd0171e35c Revert "filer store: redis module upgrade" 
This reverts commit 40161c9cec.
 2021-01-11 02:09:18 -08:00
Chris Lu
394513f598 filer: ensure seamless meta data updates 2021-01-11 02:08:55 -08:00
Chris Lu
6e12a3a490 refactoring 2021-01-11 02:08:26 -08:00
Chris Lu
62ee484d12 refactoring 2021-01-11 00:03:13 -08:00
Chris Lu
1efb51ba84 filer: change to saveToFilerLimit from cacheToFilerLimit 
short circuit saving small files to volume server
 2021-01-10 23:14:46 -08:00
Chris Lu
f0d3b3bf93 2.20 2021-01-09 22:55:22 -08:00
Chris Lu
d3d3f2fb9b mount: default to 128 concurrent writers 2021-01-09 22:53:37 -08:00
Chris Lu
90b117acf1 update ccache version 2021-01-08 02:17:43 -08:00
Chris Lu
e2e0e30a51 filer: redis default to false for useReadOnly and routeByLatency 2021-01-08 01:12:44 -08:00
Chris Lu
40161c9cec filer store: redis module upgrade 2021-01-08 01:08:44 -08:00
Chris Lu
30efaa66c6 server: add option to disable master 2021-01-07 02:35:47 -08:00
Chris Lu
07f4703bfc
Merge pull request #1738 from qieqieplus/rocksdb 
fix rocksdb crash when list directory
 2021-01-07 02:26:41 -08:00
qieqieplus
0f18592315 alter style 2021-01-07 18:21:48 +08:00
qieqieplus
c7e7431fd3 fix seek lastkey may reach EOF 2021-01-07 17:07:56 +08:00
Chris Lu
e327385644 go fmt 2021-01-06 04:22:00 -08:00
Chris Lu
2b76854641 add "weed filer.cat" to read files directly from volume servers 2021-01-06 04:22:00 -08:00
Chris Lu
95ecf0c72f
Merge pull request #1735 from qieqieplus/rocksdb 
ignore decode error for non-entry data
 2021-01-06 02:22:18 -08:00
qieqieplus
0764fccde7 ignore decode error for non-entry data 2021-01-06 17:42:43 +08:00
Chris Lu
ab57767d4b
Merge pull request #1730 from qieqieplus/rocksdb 
impl: TTL per entry for rocksdb; fix package name
 2021-01-05 13:24:45 -08:00
qieqieplus
b2e50f602f
fix time comparison 2021-01-05 19:36:40 +08:00
qieqieplus
bbae13e415 impl: TTL per entry for rocksdb; fix package name 2021-01-05 17:58:40 +08:00
henry
b42c5b71c1 Seaweed custom header are not visible to Vue or javascript 2021-01-05 13:01:29 +08:00
henry
1309dfe612 Seaweed custom header are not visible to Vue or javascript 2021-01-05 12:58:46 +08:00
henry
942a30f04c remove sort 2021-01-05 12:24:32 +08:00
henry
a3c432dadc Merge branch 'master' of https://github.com/fuyouyshengwu/seaweedfs 2021-01-05 11:17:29 +08:00
qieqieplus
ebb223c190 fix for rocksdb kv 2021-01-04 21:21:46 +08:00
chenqieqie
40b3207f18 fix #1726 2021-01-04 18:48:55 +08:00
henry
97a94eddab Merge branch 'master' of https://github.com/fuyouyshengwu/seaweedfs 2021-01-04 14:50:01 +08:00
Chris Lu
2ce86f308e 2.19 2021-01-03 17:24:02 -08:00
Chris Lu
9c9ba3c209 nil related 
related to https://github.com/chrislusf/seaweedfs/issues/1676
 2021-01-03 12:25:58 -08:00
Chris Lu
4b5adc60b4 upload result changes to the right output variable 2021-01-03 01:44:22 -08:00
Chris Lu
039ad101b1 manifest file also need to detect compression 
fix https://github.com/chrislusf/seaweedfs/issues/1724#issuecomment-753585046
 2021-01-03 01:42:21 -08:00
Chris Lu
5f538c5f9d Merge branch 'add_rocksdb' 2021-01-03 00:48:56 -08:00
Chris Lu
bdd166c031 adjust readme 2021-01-03 00:47:58 -08:00
Chris Lu
a04d8927a8 add scaffold 2021-01-03 00:44:52 -08:00
Chris Lu
4250a3a675 upload: add read default collection from master 
fix https://github.com/chrislusf/seaweedfs/issues/1724#issuecomment-753576071
 2021-01-02 23:43:01 -08:00
Chris Lu
f4d87f8b9d works 
performance seems about the same as leveldb for insert entries
 2021-01-02 23:32:58 -08:00
Chris Lu
2c3c2c27d7 separate prefix from namePattern 
fix https://github.com/chrislusf/seaweedfs/issues/1722
 2021-01-01 20:23:23 -08:00
Peter Cai
cd2a9865ed filer: respect Content-Type on PUT 
This is needed for the S3 gateway to conform to the S3 protocol -- the
server should always respect the Content-Type set by client.
 2020-12-31 11:32:45 +08:00
Chris Lu
3433accb1b weed upload: add TTL 2020-12-30 15:46:07 -08:00
henry
14ddd15508 1 Distinguish between folders and common files, folders are listed first 
2 Add some custom headers that need to be exposed
 2020-12-31 00:03:22 +08:00
Chris Lu
adef4ddc87 mount: load security keys 
fix https://github.com/chrislusf/seaweedfs/issues/1714
 2020-12-29 11:51:38 -08:00
Chris Lu
89977123b6 shell: add shell.toml support 
fix https://github.com/chrislusf/seaweedfs/issues/1664
 2020-12-28 15:07:18 -08:00
Chris Lu
d2ead72918 2.18 2020-12-28 12:37:10 -08:00
Chris Lu
2e6cf541e7 filer: avoid deletion error when item is not found 2020-12-28 00:18:37 -08:00
Chris Lu
6d2c979999 s3: fix regression 
fix https://github.com/chrislusf/seaweedfs/issues/1707
 2020-12-27 21:09:45 -08:00
Chris Lu
90df52d001 s3: add command option "allowEmptyFolder" to filer and server 2020-12-27 21:09:06 -08:00
Chris Lu
da7e5aaa65 filer: optimize for less number of directory lookup 
bottom up directory lookup
 2020-12-27 20:46:17 -08:00
henry
7458ff9523 Solve the Chinese name disorder of download file 2020-12-28 11:28:33 +08:00
Chris Lu
561a2ff0bc s3: add option for "alllowEmptyFolder" 2020-12-27 10:50:27 -08:00
Chris Lu
aa020ee7e7 s3: restore V2 auth type 
The added headers were included in calculating the signatures, failing the comparison
 2020-12-26 22:01:16 -08:00
Chris Lu
da87f6b265 remove unused code 2020-12-26 15:21:12 -08:00
Chris Lu
0a7c5f85a9 filer: add namePattern to search in current folder 2020-12-26 15:05:31 -08:00
Chris Lu
515b7632d7 2.17 2020-12-25 23:58:22 -08:00
Chris Lu
0a067944cc filer: add retries during volume moving 
fix https://github.com/chrislusf/seaweedfs/issues/1704
 2020-12-25 02:32:55 -08:00
Chris Lu
090f85be4b s3: support config action Admin:bucket 2020-12-25 00:38:56 -08:00
Chris Lu
75613b2cc7 Merge branch 'add_hbase' 2020-12-24 12:13:05 -08:00
Chris Lu
0e016bc7bd hbase add ttl 2020-12-24 12:10:35 -08:00
Chris Lu
8e48a235e2 s3: avoid looping if the directory is empty 
fix https://github.com/chrislusf/seaweedfs/issues/1701
 2020-12-24 11:34:52 -08:00
Chris Lu
1620de08ae added more logs 2020-12-24 01:52:06 -08:00
Chris Lu
a09cd28986 add debug messages 2020-12-24 01:42:15 -08:00
Chris Lu
75c6edba9e filer: hbase add enabled flag 2020-12-24 00:19:16 -08:00
Chris Lu
9bf6c10505 fix prefix 2020-12-24 00:18:59 -08:00
Chris Lu
2fea8cfa0f fix compilation 2020-12-23 23:54:15 -08:00
Chris Lu
53bc1ea25b fix compilation 2020-12-23 23:53:46 -08:00
Chris Lu
b5e2be635a adjust for directory listing 2020-12-23 23:49:22 -08:00
Chris Lu
c4a202ec41 fix wrong column family 2020-12-23 23:23:05 -08:00
Chris Lu
c3d1b3b5aa hook up 2020-12-23 21:49:01 -08:00
Chris Lu
94e3757c08 fix 2020-12-23 21:45:16 -08:00
Chris Lu
64c48c9724 seems compiling 2020-12-23 21:44:53 -08:00
Chris Lu
15da5834e1 Merge branch 'master' into support_ssd_volume 2020-12-23 14:37:37 -08:00
Chris Lu
3be3635799 just log error when checking empty dir failed 2020-12-23 14:36:48 -08:00
Chris Lu
0ca9d89589 s3: break loop if error 
fix #1701
 2020-12-23 14:34:59 -08:00
Chris Lu
132f275d04 s3: log errors instead of stopping when checking isDirectoryAllEmpty has error 2020-12-23 12:21:21 -08:00
Chris Lu
db62090f88 s3: break loop if error 
fix https://github.com/chrislusf/seaweedfs/issues/1701
 2020-12-23 12:14:51 -08:00
Baptiste Mille-Mathias
385a4dec30 [shell] Clarify bucket replication setting order 2020-12-23 09:35:24 +01:00
Chris Lu
1c7e1295dc Merge branch 'master' into support_ssd_volume 2020-12-22 17:44:52 -08:00
Chris Lu
da134a2eb7 minor 2020-12-22 17:43:13 -08:00
Chris Lu
beb3b8ddb0 shell: change bucket.list to s3.bucket.list, same for create and delete 2020-12-22 17:40:55 -08:00
Chris Lu
4cf3176cbb filer first, master second 2020-12-22 17:34:44 -08:00
Chris Lu
a1009e8044 filer: add -defaultStoreDir so that filer.toml can be skipped 
fix https://github.com/chrislusf/seaweedfs/issues/1659
 2020-12-22 17:33:40 -08:00
Chris Lu
a0990b929d minor 2020-12-22 17:33:40 -08:00
Baptiste Mille-Mathias
0ed5345954 [shell] Add more information to bucket.create 
State the replication setting that will be honor if not defined.
 2020-12-22 17:18:59 +01:00
Chris Lu
97e3432dfe avoid wrong error 
fix https://github.com/chrislusf/seaweedfs/issues/1691
 2020-12-22 02:46:24 -08:00
Chris Lu
4f31c1bb94 go fmt 2020-12-22 02:34:08 -08:00
Chris Lu
6c4f32d173 remove unused code 2020-12-22 02:33:31 -08:00
Chris Lu
90d785a15f filer: redis, redis cluster, cassandra support super large directory 2020-12-22 02:26:05 -08:00
Chris Lu
fe46411cd4 handle directory listing edge cases 2020-12-21 23:37:43 -08:00
Chris Lu
c74bede730 handle listing directories 2020-12-21 23:19:05 -08:00
Chris Lu
adf8cb4000 filer: path-specific stores trim out common prefixes 2020-12-21 22:57:13 -08:00
Chris Lu
0823bde8d8 refactor: split into two files 2020-12-21 22:05:15 -08:00
Chris Lu
8ab70ccf76 refactor 2020-12-21 21:46:32 -08:00
Chris Lu
65cc6dc636 refactor to dedicated function 2020-12-21 21:18:34 -08:00
Chris Lu
488c2680e8 change default filer store root 2020-12-21 21:15:22 -08:00
Chris Lu
f5a19a87be filer: report error if the path specific store is not found 
related https://github.com/chrislusf/seaweedfs/issues/1691
 2020-12-21 12:43:52 -08:00
Chris Lu
6912bf94ae 2.16 2020-12-20 15:30:12 -08:00
Chris Lu
41c0f3ad24 filer: support path-specific filer store 2020-12-19 01:27:09 -08:00
Chris Lu
2260864b45 a cleaner fix 2020-12-18 10:39:30 -08:00
Chris Lu
b7fef8f693 fix when a volume server has only hdd volumes 2020-12-18 10:34:16 -08:00
Chris Lu
0d5683fb0e todo: load path-specific store from filer.toml 2020-12-18 03:05:42 -08:00
Chris Lu
3269fd7eaf filer: use store by path 2020-12-18 02:57:49 -08:00
Chris Lu
e605f1e001 refactor, add getActualStore() by path 2020-12-18 02:55:00 -08:00
Chris Lu
23903aa95b refactoring 2020-12-18 02:35:45 -08:00
Chris Lu
4e6b316913 assert FilerStoreWrapper is VirtualFilerStore 2020-12-18 02:17:06 -08:00
Chris Lu
9abf016af1 Merge branch 'master' into support_ssd_volume 2020-12-17 22:30:19 -08:00
Chris Lu
738c4fd203 fix logging when error 
address https://github.com/chrislusf/seaweedfs/pull/1685#issuecomment-747845601
 2020-12-17 19:37:46 -08:00
Chris Lu
d9e8479c06 adjust UI max count 2020-12-17 13:47:51 -08:00
Chris Lu
3cdf5945a2 adjust UI 2020-12-17 13:37:00 -08:00
Chris Lu
f696a2b2a7 assign volumes based on disk type 2020-12-17 13:25:05 -08:00
Chris Lu
daa8157fc2 Merge branch 'master' into support_ssd_volume 2020-12-17 13:05:20 -08:00
Chris Lu
e2076201d7 volume: avoid reprocessing the same volume 
fix https://github.com/chrislusf/seaweedfs/issues/1682
 2020-12-17 13:03:39 -08:00
Chris Lu
986cbdf7d9 Revert "Merge pull request #1683 from qieqieplus/master" 
This reverts commit 8cb67952db, reversing
changes made to 200e56215a.
 2020-12-17 12:46:42 -08:00
Chris Lu
f56e6d231e Revert "Merge pull request #1683 from qieqieplus/master" 
This reverts commit 8cb67952db, reversing
changes made to 200e56215a.
 2020-12-17 12:46:20 -08:00
Chris Lu
1bf22c0b5b go fmt 2020-12-16 09:14:05 -08:00
Chris Lu
1d88865869 passing disk type along 2020-12-16 09:10:14 -08:00
Chris Lu
23014b6810 Merge branch 'master' into support_ssd_volume 2020-12-16 08:26:51 -08:00
Chris Lu
b789767cd1 filer: return http status 499 for client closed connection 
fix https://github.com/chrislusf/seaweedfs/issues/1684
 2020-12-16 08:18:00 -08:00
chenqieqie
45f902a9b7 smaller critical section 2020-12-16 20:23:51 +08:00
chenqieqie
4e58a4f24e fix race condition when loading volumes concurrently 2020-12-16 18:49:10 +08:00
Chris Lu
3c7f9633eb show error a little bit more obviously 
fix https://github.com/chrislusf/seaweedfs/issues/1678
 2020-12-14 09:39:33 -08:00
Chris Lu
97dcae827b Merge branch 'master' into support_ssd_volume 2020-12-14 01:27:31 -08:00
Chris Lu
3d47c38262 collection.delete requires _default_ as the default empty collection name 
fix https://github.com/chrislusf/seaweedfs/issues/1677
 2020-12-14 01:05:20 -08:00
Chris Lu
2e8dba571b adjust volume server UI 2020-12-14 00:51:57 -08:00
Chris Lu
bf5873022d Merge branch 'master' into support_ssd_volume 2020-12-14 00:32:17 -08:00
Chris Lu
b7e3ca9172
Merge pull request #1650 from kmlebedev/masterServerEnableUI 
security master use access.ui
 2020-12-14 00:31:51 -08:00
Chris Lu
021358749b better detect IP address 
following advice from https://github.com/chrislusf/seaweedfs/issues/1671
 2020-12-14 00:30:20 -08:00
Konstantin Lebedev
c4459249ef ui interface is always on 2020-12-14 13:24:41 +05:00
Chris Lu
5cffddebb9 add logging 2020-12-14 00:11:52 -08:00
Chris Lu
933f2f4cfd find location with matching disk type 2020-12-13 23:39:00 -08:00
Chris Lu
94525aa0fd allocate volume by disk type 2020-12-13 23:08:21 -08:00
Chris Lu
f6a419c26c disk type configurable for each folder 2020-12-13 22:49:56 -08:00
Chris Lu
7c52a35942 refactor 2020-12-13 22:29:52 -08:00
Chris Lu
c45d6a865b Merge branch 'master' into support_ssd_volume 2020-12-13 22:14:18 -08:00
Chris Lu
38fc89d041 filer store: sql update if any insert error happens 
fix https://github.com/chrislusf/seaweedfs/issues/1673
 2020-12-13 20:49:44 -08:00
Chris Lu
23280257df sql put kv: avoid unnecessary update 2020-12-13 19:45:47 -08:00
Chris Lu
a9db24cd05 master allocate volumes if ssd type runs out 2020-12-13 19:44:57 -08:00
Chris Lu
ac4d4a65af Update Makefile 2020-12-13 12:34:58 -08:00
Chris Lu
8baba93fce rename parameter name to "disk" 2020-12-13 12:06:21 -08:00
Chris Lu
51eadaf2b6 rename parameter name to "disk" 2020-12-13 12:05:31 -08:00
Chris Lu
0d2ec832e2 rename from volumeType to diskType 2020-12-13 11:59:32 -08:00
Chris Lu
715b199eeb fix tests 2020-12-13 04:14:50 -08:00
Chris Lu
66cd219b15 Merge branch 'master' into support_ssd_volume 2020-12-13 04:08:36 -08:00
Chris Lu
813453f3e1 2.15 2020-12-13 03:45:35 -08:00
Chris Lu
17372ac3da balance ssd volumes 2020-12-13 03:40:33 -08:00
Chris Lu
d156c74ec0 volume server set volume type and heartbeat to the master 2020-12-13 03:11:24 -08:00
Chris Lu
e9cd798bd3 adding volume type 2020-12-13 00:58:58 -08:00
Chris Lu
16cd6fb278 fix variable name 2020-12-12 21:19:40 -08:00
Chris Lu
5c465293e9 correctly determine whether a folder is empty 
avoid edge cases that deleting the folder if previous 32 directories are all empty

early terminate if one file is found
 2020-12-12 16:19:29 -08:00
Chris Lu
14910d035c the pagination size was too big for recursive deletion 2020-12-12 13:26:10 -08:00
Chris Lu
f930c713fc more efficient recursion 2020-12-12 13:25:19 -08:00
Chris Lu
316d1b4e69 refactor APIs 2020-12-12 12:42:53 -08:00
Chris Lu
03637d6f57 s3: move "delete-directory-if-empty" to read time 
move "delete-directory-if-empty" to read time instead of entry deletion time

the listing speed for a s3 bucket folder will slow down if it has many sub folders

related to 0d345ac97d

fix https://github.com/chrislusf/seaweedfs/issues/1647

fix https://github.com/chrislusf/seaweedfs/issues/1670
 2020-12-12 03:38:34 -08:00
Chris Lu
37075a414d adjust logs 2020-12-12 03:33:57 -08:00
Chris Lu
ae655033ac adjust logging 2020-12-11 16:57:53 -08:00
Chris Lu
d986c7196d use append time instead of filer's own modification time 
fix https://github.com/chrislusf/seaweedfs/issues/1669
 2020-12-11 16:55:18 -08:00
Chris Lu
3fedfec1e7 check cross device rename error 2020-12-10 23:50:32 -08:00
Chris Lu
c2f18a10cb minor 2020-12-10 22:23:22 -08:00
Chris Lu
4b0c2a846b add logs 2020-12-10 19:55:28 -08:00
Chris Lu
83078ac6ce filer: change to /etc/seaweedfs folder on filer 
fix https://github.com/chrislusf/seaweedfs/issues/1666
 2020-12-10 11:11:02 -08:00
Chris Lu
26731694f8 s3: use static configuration by default 
So that users can still use the previous configuration files.

If leave it empty, s3 will try to use the version from filer
 2020-12-10 00:59:04 -08:00
Chris Lu
765b3ef1be save /etc/iam/identity.json inside filer store 2020-12-10 00:15:22 -08:00
Chris Lu
97c942b5ee paginate through large directories 2020-12-09 23:23:38 -08:00
Chris Lu
02a160c3fd s3: recursively iterate all sub folders 
fix https://github.com/chrislusf/seaweedfs/issues/1656
 2020-12-09 16:47:34 -08:00
Chris Lu
a9c619c270 added error handling for saving 2020-12-09 01:34:24 -08:00
Chris Lu
4211601eab set file handle reader to nil 2020-12-08 22:48:18 -08:00
Chris Lu
b52ae9cef8 add s3 debug 2020-12-08 22:27:04 -08:00
Chris Lu
8e78187a97 add back last read chunk cache to reader and properly close the reader 2020-12-08 22:26:46 -08:00
Chris Lu
900d22c6ec mount: avoid memory leaking read buffer 
fix https://github.com/chrislusf/seaweedfs/issues/1654

the reader goes together with the file handle, which may stay for a long time.
 2020-12-08 02:38:53 -08:00
Chris Lu
06bb7bf6c0 adding /usr/local/etc/seaweedfs for bsd style config search path 
related to https://reviews.freebsd.org/D27391
 2020-12-07 16:46:48 -08:00
Chris Lu
fce8803087 break import cycle 2020-12-07 00:29:17 -08:00
Chris Lu
606051cc79 filer: cache "/etc" filder small files to filer store 2020-12-07 00:11:35 -08:00
Chris Lu
ac22f1dd26 filer subscribe: handle rename subscription 2020-12-07 00:10:49 -08:00
Chris Lu
eed87791b7 s3: subscribe to s3.configure changes 2020-12-07 00:10:29 -08:00
Chris Lu
01e2da5782 refactoring 2020-12-06 23:16:20 -08:00
Chris Lu
77286f8bea fix test 2020-12-06 21:56:13 -08:00
Chris Lu
57578a6cd1 change parameter help message 2020-12-06 21:54:55 -08:00
Chris Lu
78f3ab439c refactoring 2020-12-06 21:54:34 -08:00
Chris Lu
9acda432fe fix import cycle 2020-12-06 20:12:52 -08:00
Chris Lu
ae5eb85a06 refactoring 2020-12-06 20:05:06 -08:00
Chris Lu
5931a2f53f filer conf: support json conf only 2020-12-06 19:47:06 -08:00
Chris Lu
4c72482536 shell: fix fs.configure 2020-12-06 19:43:31 -08:00
Chris Lu
46b91228af 2.14 2020-12-06 18:56:58 -08:00
Chris Lu
263eb29e9f filer: add option to cache small files to filer store 2020-12-04 22:39:43 -08:00
Chris Lu
d171d9f988 volume: ensure the volume dat and idx files are always closed 
fix https://github.com/chrislusf/seaweedfs/issues/1646
 2020-12-04 21:50:26 -08:00
Konstantin Lebedev
eadbba5c95 security master use access.ui 2020-12-04 12:57:59 +05:00
Chris Lu
cc839f935d better unit detection 2020-12-03 19:23:59 -08:00
Chris Lu
b219ccfe68 Update volume_ttl_test.go 2020-12-03 12:55:44 -08:00
Chris Lu
a83b8e7b01 ttl fix: 3y was converted into 96m 2020-12-03 10:53:29 -08:00
Chris Lu
5fb60b7135
Merge pull request #1596 from kmlebedev/store_s3cred 
S3 credentials store in filer
 2020-12-03 00:40:20 -08:00
Konstantin Lebedev
14699dfcef use content field of entry 2020-12-02 17:19:05 +05:00
Konstantin Lebedev
a3d4b50a49 use entry content filed 2020-12-02 16:12:13 +05:00
Chris Lu
003b6245e7 fix nil 2020-12-02 00:09:19 -08:00
Chris Lu
f3bb645018 file open error 2020-12-01 23:37:49 -08:00
Chris Lu
dc0bc48257 return file open error 2020-12-01 23:36:49 -08:00
Chris Lu
45dba088cd fix tests 2020-12-01 19:37:21 -08:00
Chris Lu
04062c56c7 webdav: improve webdav upload speed 2020-12-01 15:32:27 -08:00
Konstantin Lebedev
03620776ec Merge branch 'upstream_master' into store_s3cred 2020-12-01 16:03:34 +05:00
Chris Lu
005a6123e9
Merge pull request #1643 from hilimd/master 
fix tls grpc ca path
 2020-11-30 17:16:46 -08:00
Chris Lu
f4abd01adf filer: cache small file to filer store 2020-11-30 04:34:04 -08:00
Chris Lu
a9c6be5fc3 set creation time 2020-11-30 03:11:52 -08:00
Chris Lu
141ce67c09 close http request body 2020-11-30 02:45:00 -08:00
Chris Lu
0e99531dbf 2.13 2020-11-29 17:00:03 -08:00
Chris Lu
965413c21b shell: add volume.vacuum command 2020-11-28 23:18:02 -08:00
Chris Lu
96c48bc8a8 fix test 2020-11-28 03:03:43 -08:00
Chris Lu
c57a7839ec adjust comments 2020-11-28 03:01:03 -08:00
Chris Lu
8438fb0cc3 fix ec shard spreading error 2020-11-28 02:21:16 -08:00
Chris Lu
536202e4e0 minor 2020-11-28 02:16:06 -08:00
Chris Lu
99c4e50d3d minor 2020-11-28 00:14:11 -08:00
Chris Lu
2c913dde04 volume: detect and drop volumes with disk IO error 
from Jethro in slack:
is it possible to make the assign request a bit smarter? Currently I’m in the state that a disk failed but all assign request are being send to this volume. It would be cool if the master sees this and stopped using this volume.
e=HTTP(http://x:8089/913,045a782b63176edf) not 200 but 500 Internal Server Error
Body={"size":740167,"error":"failed to write to local disk: write /mnt/v9/913.dat: input/output error","eTag":"ee4381e202212ff3aee647704c036689"}
e=HTTP(http://x:8089/913,045a782c90240077) not 200 but 500 Internal Server Error
Body={"size":792779,"error":"failed to write to local disk: write /mnt/v9/913.dat: input/output error","eTag":"c43463ccc11eb6eb2fc306f407a6a953"}
e=HTTP(http://x:8089/913,045a782e6b7901ea) not 200 but 500 Internal Server Error
Body={"size":3962392,"error":"failed to write to local disk: write /mnt/v9/913.dat: input/output error","eTag":"04c91198e9b276c81f11dbf189af5d28"}
 2020-11-28 00:09:29 -08:00
Chris Lu
9ac4935f22 read from volume index file directly instead of open a separate file 
fix https://github.com/chrislusf/seaweedfs/issues/1640

read from volume index file directly instead of open a separate file,
to ensure reading latest index entries.
 2020-11-27 16:18:48 -08:00
Chris Lu
85554bea38 filer: readonly handle static resources 
fix https://github.com/chrislusf/seaweedfs/issues/1641
 2020-11-27 13:42:14 -08:00
Chris Lu
6d30b21b10 volume: add "-dir.idx" option for separate index storage 
fix https://github.com/chrislusf/seaweedfs/issues/1265
 2020-11-27 03:17:10 -08:00
Chris Lu
3c229eb677 refactor 2020-11-26 17:21:55 -08:00
Chris Lu
c0fb4a3766 add comments 2020-11-26 15:22:42 -08:00
Chris Lu
6783ae5f8f rename 2020-11-26 15:19:43 -08:00
Chris Lu
983fd90010 marked as WIP 2020-11-26 15:08:52 -08:00
Chris Lu
97f706f35b just refactoring 2020-11-26 15:08:05 -08:00
Chris Lu
b11449f955 pass in dir.idx parameter to DiskLocation 2020-11-26 14:59:03 -08:00
Chris Lu
0d345ac97d s3: remove empty parent folder on delete 
fix https://github.com/chrislusf/seaweedfs/issues/1637
 2020-11-26 12:21:58 -08:00
Chris Lu
cc2bd97ad9 refactor 2020-11-26 11:25:56 -08:00
Chris Lu
0da7ecfd29 go fmt 2020-11-26 11:22:30 -08:00
Chris Lu
effa00ed08 refactor 2020-11-26 11:21:58 -08:00
Chris Lu
513bcd6e0d filer: avoid duplicated FindEntry for deletion 2020-11-26 11:14:56 -08:00
Chris Lu
1ae108efca add debug message 2020-11-26 10:41:10 -08:00
Chris Lu
a102157020 do not retry if failed with a non transport related error 2020-11-26 01:29:30 -08:00
Konstantin Lebedev
4e55baf5b1 s3 config changes 2020-11-26 03:50:53 +05:00
Konstantin Lebedev
52c8f2fc9a s3iam test 2020-11-26 02:26:45 +05:00
Konstantin Lebedev
a26f1b2040 new pkg s3iam 2020-11-26 01:30:11 +05:00
Konstantin Lebedev
6206737df2 s3 configure 2020-11-25 21:02:31 +05:00
Chris Lu
eab53ea80d filer leveldb store: a bit more efficient directory listing with prefix 2020-11-22 21:10:41 -08:00
Chris Lu
d3be2ceedd 2.12 2020-11-22 17:17:33 -08:00
Chris Lu
c7ebadc25d avoid possible concurrent access inside ensureCorrectWritables() 2020-11-22 17:15:59 -08:00
hilimd
9aa990f80f
Merge pull request #39 from chrislusf/master 
sync
 2020-11-22 20:29:17 +08:00
limd
e8296104fc fix tls grpc ca path 2020-11-22 20:27:15 +08:00
Chris Lu
92f906b6fc remove zstd 
fix https://github.com/chrislusf/seaweedfs/issues/1629
 2020-11-21 13:06:45 -08:00
Aleksandr Demshin
176950ea78 fix typo 2020-11-20 19:50:46 +05:00
Chris Lu
7385103057 fix volume placement validating 
fix https://github.com/chrislusf/seaweedfs/issues/1626
 2020-11-20 01:12:25 -08:00
Konstantin Lebedev
27e73de797 Merge branch 'upstream_master' into store_s3cred 
# Conflicts:
#	weed/s3api/filer_util.go
 2020-11-19 18:16:44 +05:00
Chris Lu
290b5e2cd0 directly delete file chunks 
keeping current async deletions for now
 2020-11-17 17:20:21 -08:00
Chris Lu
781585b195 remove unnecessary checking 2020-11-17 17:00:06 -08:00
Chris Lu
8cb8cd4cc5 add locks 2020-11-17 16:59:48 -08:00
Chris Lu
7c9e592c2d s3: avoid nil resp when having error 
fix https://github.com/chrislusf/seaweedfs/issues/1622
 2020-11-17 11:23:17 -08:00
Chris Lu
dc304342b2 fs.configure: configurable volume growth 2020-11-17 01:00:02 -08:00
Chris Lu
b04375a3d1 use alternative assign request 2020-11-17 00:36:21 -08:00
Chris Lu
9add554feb fail fast if user configures collection for a bucket 2020-11-16 19:57:08 -08:00
Chris Lu
6204dc6e85 add comment 2020-11-16 16:57:31 -08:00
Chris Lu
ef3e068c51 deprecating "filer.options.buckets_fsync" in filer.toml 2020-11-16 16:56:49 -08:00
Chris Lu
5f19e81dab filer confi: support hierachical configuration 2020-11-16 16:50:12 -08:00
Chris Lu
ed1ce3f299 adjust help message 2020-11-16 09:54:26 -08:00
Chris Lu
c0d279c54e filere.conf: prefer to use json format 2020-11-15 21:48:17 -08:00
Chris Lu
83527a8f55 adds more help message 2020-11-15 21:31:14 -08:00
Chris Lu
af658ea970 fix typo 2020-11-15 21:16:28 -08:00
Chris Lu
98827d68d9 2.11 2020-11-15 20:25:56 -08:00
Chris Lu
ee2fa14dbe filer conf: delete location specific configuration 2020-11-15 20:15:47 -08:00
Chris Lu
2bd6fd3bbe remove unused function 2020-11-15 20:15:07 -08:00
Chris Lu
71056dae07 fs.configure: read and local add filer configuration 2020-11-15 18:09:35 -08:00
Chris Lu
0ea5c087ce go fmt 2020-11-15 16:59:28 -08:00
Chris Lu
95c0de285d refactoring 2020-11-15 16:58:48 -08:00
Chris Lu
500bcab953 refactoring 2020-11-15 14:41:56 -08:00
Chris Lu
590f02179d filer: load filer conf when starting 2020-11-15 14:06:03 -08:00
Chris Lu
0a406f652e load filer conf and match by prefix 2020-11-15 00:26:05 -08:00
Chris Lu
68043cfcac add reference implementation to detect create/update/delete/rename events 2020-11-14 21:21:58 -08:00
Chris Lu
0fc9ffc603 able to read filer.conf if updated 2020-11-14 21:21:20 -08:00
Chris Lu
442e092995 filer watch "/etc" folder for configuration changes 2020-11-14 14:26:08 -08:00
ruitao.liu
a9990a1dc6 adjust check bucket if exist or has access. 2020-11-13 17:13:20 +08:00
Chris Lu
e6333da65a enable admin to access all buckets 2020-11-12 13:57:54 -08:00
Chris Lu
7094492428 fix ErrNotFound 2020-11-12 13:30:46 -08:00
Chris Lu
c6a0704019 rename function 2020-11-12 13:30:08 -08:00
Chris Lu
ed3b0f81c6
Merge pull request #1613 from taozix/master 
check permission for bucket delete/head.
 2020-11-12 13:25:28 -08:00
ruitao.liu
c4f0fd6e1b skip if entry.Extended map is nil. 2020-11-12 17:59:31 +08:00
ruitao.liu
ab966410d2 return NoSuchBucket instead of InternalError delete non-existed bucket. 2020-11-12 16:44:16 +08:00
Chris Lu
a4dfa00030 adjust filer conf 2020-11-12 00:39:59 -08:00
Chris Lu
2251a8e472 minor 2020-11-12 00:38:58 -08:00
Chris Lu
5a16f17e47 remove unused message type 2020-11-12 00:38:23 -08:00
ruitao.liu
e06676f007 check permission for bucket delete/head. 2020-11-12 16:15:59 +08:00
Chris Lu
c32b6f4d9d prepare for path based configuration 2020-11-11 23:46:45 -08:00
Chris Lu
0fbc27b640
Merge pull request #1612 from taozix/master 
add owner attr for bucket.
 2020-11-11 22:19:47 -08:00
ruitao.liu
d7cc0498e0 check if bucket already exists. 2020-11-12 14:11:03 +08:00
Chris Lu
827930e249
Merge pull request #1610 from kmlebedev/prefer_read_in_this_dc 
prefer to read from volumes in this data center
 2020-11-11 22:09:41 -08:00
Konstantin Lebedev
0880aff224 update dc flag desc 2020-11-12 10:07:52 +05:00
ruitao.liu
dbba8cb57e fix if nil map in entry. 2020-11-12 11:50:19 +08:00
Konstantin Lebedev
aa30604cec fix tests 2020-11-12 08:49:38 +05:00
Konstantin Lebedev
1eec5c8d5d gen pb 2020-11-12 04:10:06 +05:00
Konstantin Lebedev
fc7baef5bb fiil serverUrls sorted by data center 2020-11-12 02:13:33 +05:00
Chris Lu
745ee8d8f3 avoid unnecessary error message 
fix https://github.com/chrislusf/seaweedfs/issues/1611
 2020-11-11 13:00:05 -08:00
Chris Lu
a29abd8600 add a {X-,}Idempotency-Key header for http POST 
fix https://github.com/chrislusf/seaweedfs/issues/1592

according to https://github.com/golang/go/issues/19943
 2020-11-11 12:56:53 -08:00
Chris Lu
8708e7a64d print out data size 2020-11-11 12:51:44 -08:00
Chris Lu
c6bd244ebd add TODO 2020-11-11 12:51:27 -08:00
Konstantin Lebedev
b73ef6aa95 Merge remote-tracking branch 'upstream/master' into prefer_read_in_this_dc 2020-11-11 23:29:57 +05:00
Konstantin Lebedev
dc26012a3b initial 2020-11-11 15:03:47 +05:00
Chris Lu
86cdb2a3e5 adjust logs 2020-11-11 02:01:24 -08:00
Chris Lu
73f934d5de s3: do not close reader too early 
fix https://github.com/chrislusf/seaweedfs/issues/1609
 2020-11-11 01:49:39 -08:00
ruitao.liu
5b636b3242 Add bucket owner attr. 2020-11-11 16:20:59 +08:00
Chris Lu
15956d8a8e 2.10 2020-11-10 23:17:13 -08:00
Chris Lu
de3bdd0651 delete old volume replica 
related to https://github.com/chrislusf/seaweedfs/issues/1607

old is:
* older compaction revision
* older modified time
* smaller volume size
 2020-11-10 12:26:05 -08:00
Chris Lu
1dd3a6ac36 avoid verbose logs 2020-11-09 15:56:11 -08:00
Chris Lu
6856b0d57e filer: add API to add/modify/delete tagging 2020-11-09 01:00:07 -08:00
Konstantin Lebedev
6cd6ff6962 avoid blank response for 404 2020-11-04 22:54:47 +05:00
Chris Lu
0dafcf1f5a volume: detect max volume count changes based on disk usage 
fix https://github.com/chrislusf/seaweedfs/issues/1594
 2020-11-03 14:43:17 -08:00
Chris Lu
06ee199ef3 ensure entryViewCache is in sync with the chunks 2020-11-03 12:22:19 -08:00
Chris Lu
b132cb893f in case some chunks modified earliar are added later 2020-11-03 12:22:19 -08:00
Chris Lu
0ae4d7bfdf lock "addChunks()" 2020-11-03 12:22:19 -08:00
Konstantin Lebedev
e1190b3224 load S3 config from filer 
https://github.com/chrislusf/seaweedfs/issues/1500
 2020-11-03 21:45:56 +05:00
Chris Lu
d29b787220 store together with "Seaweed-" prefix 2020-11-03 00:21:10 -08:00
Chris Lu
5448781dfc filer: Headers with "Seaweed-" prefix are stored as extended properties 2020-11-03 00:15:51 -08:00
Chris Lu
0ecff9521d avoid []byte reuse 2020-11-02 23:27:54 -08:00
Chris Lu
031aa70da6 Revert "revert bytebufferpool, seems problematic" 
This reverts commit e0fcab47c0.
 2020-11-02 19:11:25 -08:00
Chris Lu
7d5db73d98 go fmt 2020-11-02 14:21:07 -08:00
Chris Lu
a9662d44ac avoid closing chunkSaveErrChan too early 2020-11-02 14:20:38 -08:00
Chris Lu
e0fcab47c0 revert bytebufferpool, seems problematic 2020-11-02 13:49:38 -08:00
Chris Lu
e1423a83dd mount: clear possible leftover entry data in memory 2020-11-01 23:28:56 -08:00
Chris Lu
9b10d1ac33 2.08 2020-11-01 12:45:20 -08:00
Chris Lu
8750cac090 move to util.RetryWaitTime 2020-11-01 02:36:43 -08:00
Chris Lu
ef908e166b break import cycle 2020-11-01 02:28:11 -08:00
Chris Lu
4609644658 mount: remove files that may still are open 
related to https://github.com/chrislusf/seaweedfs/issues/1581
 2020-11-01 01:33:26 -08:00
Chris Lu
df8d976bb0 refactoring 2020-11-01 01:58:48 -07:00
Chris Lu
7192a378cc add retry to master clients also 2020-11-01 01:40:16 -07:00
Chris Lu
004aa9ca5d only retry transport related errors 2020-11-01 01:21:44 -07:00
Chris Lu
f7b21973ac mount: add retry for all operations with filer 
fix https://github.com/chrislusf/seaweedfs/issues/1589
 2020-11-01 01:11:08 -07:00
Chris Lu
de86945aeb go fmt 2020-10-31 16:45:38 -07:00
Chris Lu
306062b4e7 filer: add CORS support 2020-10-31 16:44:03 -07:00
Chris Lu
05acc33294 volume: add CORS support 2020-10-31 16:31:39 -07:00
Chris Lu
9708df47d1 ensure offset in the index file is positive 2020-10-31 15:50:01 -07:00
Chris Lu
10f9f1cc26 handle mknod requests 2020-10-30 23:51:32 -07:00
Chris Lu
8826601be1 mount: optional limit for the number of concurrent writers 2020-10-30 21:22:20 -07:00
Chris Lu
be95f68ca7 scaffold for sftpd 2020-10-30 20:57:08 -07:00
Konstantin Lebedev
d2a4ffdd60 glog 2020-10-31 03:21:54 +05:00
Konstantin Lebedev
46303c36bf When the volume server is stopped, the master server immediately sees the deletion of volumes 2020-10-31 02:48:25 +05:00
Chris Lu
eb32af25d4 make space for the QR code 2020-10-30 10:55:31 -07:00
Chris Lu
7ecbb4b3c9 clean up 2020-10-30 10:46:31 -07:00
Kenny
9cc6575ab5
Update filer_copy.go 
修改 weed filer.copy 上传目录时文件名变子目录的问题
 2020-10-30 18:16:49 +08:00
Chris Lu
c057dd1f64 add back upload retries 2020-10-30 02:16:34 -07:00
Chris Lu
19098f2c2d make a copy of the filer.entry for most read operations 2020-10-30 01:23:26 -07:00
Chris Lu
6135cbaa02 clear file.entry 2020-10-30 01:23:26 -07:00
Chris Lu
02dc51b1ed properly clear out the file.entry 2020-10-29 23:31:06 -07:00
Chris Lu
6560ac6466 volume loading: trim out unreachable idx file content 
fix https://github.com/chrislusf/seaweedfs/issues/1583
 2020-10-29 22:25:23 -07:00
Chris Lu
0b68b68ec4 reload entry only when it is a hard link 
fix https://github.com/chrislusf/seaweedfs/issues/1581
 2020-10-29 16:32:45 -07:00
Chris Lu
47047516d6 Revert "increase default volume file size limit to 1024" 
This reverts commit 0983060a

I am confused why this was changed to 1024 by myself. 1GB should be too large for most cases.
 2020-10-29 15:46:29 -07:00
Chris Lu
6f8b426f4f
Merge pull request #1578 from taozix/master 
S3 bucket list, response with uploaded storageclass.
 2020-10-29 08:32:27 -07:00
ruitao.liu
22a9ea0512 adjust s3 header file. 2020-10-29 16:05:40 +08:00
Chris Lu
c6d9974fe7 server: Add option to enable/disable volume server 
fix https://github.com/chrislusf/seaweedfs/issues/1572
 2020-10-29 00:24:18 -07:00
ruitao.liu
b917be7955 S3 bucket list, response with uploaded storageclass. 2020-10-29 14:57:19 +08:00
Konstantin Lebedev
918546bbdb add WaitForReady to grpc DialOption 2020-10-29 08:29:04 +05:00
Konstantin Lebedev
fab01f9d8d add readonly 2020-10-28 23:17:03 +05:00
Konstantin Lebedev
884db215a1 add shell command volume mark writable 2020-10-28 22:47:09 +05:00
Konstantin Lebedev
1ad1b8c4f6 collection Volume ReadOnly Count with detailed status 2020-10-28 17:38:26 +05:00
Chris Lu
326fcdd86b
Merge pull request #1569 from taozix/master 
save s3 metadata to filer.
 2020-10-28 03:27:24 -07:00
ruitao.liu
2dcc178d0a save s3 metadata to filer. 2020-10-28 18:16:05 +08:00
Chris Lu
744c5594da volume: automatically trim out unreachable entries 2020-10-28 01:14:39 -07:00
Chris Lu
3e925faddd trim out name extension 2020-10-28 01:14:05 -07:00
shibinbin
a67eb1afa1 fix: restart volumeserver sometime occurred out of memory error 2020-10-28 12:03:06 +08:00
Chris Lu
53c3aad875 volume: add a note file to avoid incomplete volume files 
fix https://github.com/chrislusf/seaweedfs/issues/1567
 2020-10-27 15:56:49 -07:00
Chris Lu
6da87720eb shorter tag prefix 2020-10-27 15:33:26 -07:00
Chris Lu
24bf142596 copy large file first 2020-10-27 15:31:15 -07:00
Chris Lu
432c6f8db3
Merge pull request #1566 from taozix/master 
return x-amz-tag-count header when GET object.
 2020-10-27 13:16:58 -07:00
Chris Lu
06c15ab35c volume: add special handling for .dat larger than 32GB 2020-10-27 13:11:56 -07:00
ruitao.liu
72f6b77dea fix tag prefix. 2020-10-27 18:01:37 +08:00
ruitao.liu
8766ca1b95 rename s3 tag prefix. 2020-10-27 17:33:24 +08:00
ruitao.liu
19026ae55d return x-amz-tag-count header when GET object. 2020-10-27 16:49:31 +08:00
Chris Lu
44921220b0 2.07 2020-10-25 22:03:46 -07:00
Chris Lu
e71463a9eb mount: invalide file cache when metadata is changed 2020-10-25 19:24:15 -07:00
Chris Lu
e219c57849 passing full path when assign volume locations 2020-10-25 15:46:29 -07:00
Chris Lu
f375b93aef renaming 2020-10-25 15:32:43 -07:00
Chris Lu
63b0fb54f7 minor 2020-10-25 14:15:53 -07:00
Chris Lu
27ba0a2be3 2.06 2020-10-25 12:36:47 -07:00
Chris Lu
147d4d23b6 filer.copy: automatically use the right bucket 2020-10-25 11:21:33 -07:00
Chris Lu
91fd311f7a fix logic to read entry or not 2020-10-25 10:33:51 -07:00
Chris Lu
10a4a628e9 refresh cached file entry from sync metadata updates 2020-10-25 09:42:50 -07:00
Konstantin Lebedev
2fb1fce8a0 avoid old values when missing read only flags 2020-10-25 16:21:40 +05:00
Chris Lu
da36abf033 go fmt 2020-10-24 20:12:04 -07:00
Chris Lu
16fae84414 fix format error 2020-10-24 20:11:31 -07:00
Chris Lu
b7b1eb51b1 default to 2 for better performance in common computer hardware 2020-10-24 20:08:26 -07:00
Chris Lu
79690b9cd8 revert changes causing wrong data 2020-10-24 20:07:19 -07:00
Chris Lu
9104cfa744 reduce locks 2020-10-24 19:40:35 -07:00
Chris Lu
e9d40b80b2 less verbose logs 2020-10-24 09:42:54 -07:00
Chris Lu
6c4fb243a7 remove limits to concurrent writers 2020-10-24 02:05:05 -07:00
Chris Lu
e0002f8dd7 check existing volumes for writable status 2020-10-24 01:34:31 -07:00
Chris Lu
19772d70d7 print for debugging 2020-10-24 00:12:02 -07:00
Chris Lu
5b92dfe302 adjust log level 2020-10-23 23:58:54 -07:00
Chris Lu
fb124f2d4a conditionally display volume server public url link 2020-10-23 23:58:46 -07:00
Chris Lu
1b17f71939 adjust election timeout to 10 seconds 2020-10-23 23:06:44 -07:00
Chris Lu
ef9c32ea0d adjust compression threshold 2020-10-23 23:05:55 -07:00
Chris Lu
0cea84e7ab make it easy with setting up multiple masters in a clcuster 2020-10-23 18:18:46 -07:00
Chris Lu
fc689319ae add volume public url 2020-10-23 18:03:23 -07:00
Chris Lu
71624858af avoid reusing cached []byte 
this logic is just to ensure 100% correct.

fuse message, data+message hearder > 512 will not be cached.
 2020-10-23 11:31:57 -07:00
Chris Lu
29c9fa2ef2 byte buffer for uploading 2020-10-23 11:29:51 -07:00
Chris Lu
2579edbc60 fix bug found by tests 2020-10-23 10:05:40 -07:00
Chris Lu
e0c8507d93 memory pool to reduce memory allocation 2020-10-23 00:41:40 -07:00
Chris Lu
b81359823f postgres: support empty user 2020-10-22 14:27:47 -07:00
Chris Lu
11716fbf6f make reading error more obvious 2020-10-22 13:33:45 -07:00
Chris Lu
d75bc62196 to read files that are quickly changing 2020-10-22 12:59:15 -07:00
Chris Lu
575d7952a1 add available resource stats 
fix https://github.com/chrislusf/seaweedfs/issues/1555
 2020-10-22 09:13:47 -07:00
Chris Lu
5179e559f7 skip empty logs 2020-10-22 00:35:48 -07:00
Chris Lu
de044e466e
Merge pull request #1553 from kmlebedev/multi_domainname 
Multi domainname
 2020-10-21 23:56:03 -07:00
Chris Lu
e73d6c9526 able to print partial file id 
address https://github.com/chrislusf/seaweedfs/issues/1552
 2020-10-21 23:48:07 -07:00
Konstantin Lebedev
a9a7005687 set desc of option 2020-10-22 11:23:00 +05:00
Chris Lu
720b1d9b88 adding locking to avoid nil VolumeLocationList 
fix panic: runtime error: invalid memory address or nil pointer dereference
Oct 22 00:53:44 bedb-master1 weed[8055]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x17658da]
Oct 22 00:53:44 bedb-master1 weed[8055]: goroutine 310 [running]:
Oct 22 00:53:44 bedb-master1 weed[8055]: github.com/chrislusf/seaweedfs/weed/topology.(*VolumeLocationList).Length(...)
Oct 22 00:53:44 bedb-master1 weed[8055]: #011/root/seaweedfs/weed/topology/volume_location_list.go:35
Oct 22 00:53:44 bedb-master1 weed[8055]: github.com/chrislusf/seaweedfs/weed/topology.(*VolumeLayout).enoughCopies(...)
Oct 22 00:53:44 bedb-master1 weed[8055]: #011/root/seaweedfs/weed/topology/volume_layout.go:376
Oct 22 00:53:44 bedb-master1 weed[8055]: github.com/chrislusf/seaweedfs/weed/topology.(*VolumeLayout).ensureCorrectWritables(0xc000111d50, 0xc000b55438)
Oct 22 00:53:44 bedb-master1 weed[8055]: #011/root/seaweedfs/weed/topology/volume_layout.go:202 +0x5a
Oct 22 00:53:44 bedb-master1 weed[8055]: github.com/chrislusf/seaweedfs/weed/topology.(*Topology).SyncDataNodeRegistration(0xc00042ac60, 0xc001454d30, 0x1, 0x1, 0xc0005fc000, 0xc00135de40, 0x4, 0xc00135de50, 0x10, 0x10d, ...)
Oct 22 00:53:44 bedb-master1 weed[8055]: #011/root/seaweedfs/weed/topology/topology.go:224 +0x616
Oct 22 00:53:44 bedb-master1 weed[8055]: github.com/chrislusf/seaweedfs/weed/server.(*MasterServer).SendHeartbeat(0xc000162700, 0x23b97c0, 0xc000ae2c90, 0x0, 0x0)
Oct 22 00:53:44 bedb-master1 weed[8055]: #011/root/seaweedfs/weed/server/master_grpc_server.go:106 +0x325
Oct 22 00:53:44 bedb-master1 weed[8055]: github.com/chrislusf/seaweedfs/weed/pb/master_pb._Seaweed_SendHeartbeat_Handler(0x1f8e7c0, 0xc000162700, 0x23b0a60, 0xc00024b440, 0x3172c38, 0xc000ab7100)
Oct 22 00:53:44 bedb-master1 weed[8055]: #011/root/seaweedfs/weed/pb/master_pb/master.pb.go:4250 +0xad
Oct 22 00:53:44 bedb-master1 weed[8055]: google.golang.org/grpc.(*Server).processStreamingRPC(0xc0001f31e0, 0x23bb800, 0xc000ac5500, 0xc000ab7100, 0xc0001fea80, 0x311fec0, 0x0, 0x0, 0x0)
Oct 22 00:53:44 bedb-master1 weed[8055]: #011/root/go/pkg/mod/google.golang.org/grpc@v1.29.1/server.go:1329 +0xcd8
Oct 22 00:53:44 bedb-master1 weed[8055]: google.golang.org/grpc.(*Server).handleStream(0xc0001f31e0, 0x23bb800, 0xc000ac5500, 0xc000ab7100, 0x0)
Oct 22 00:53:44 bedb-master1 weed[8055]: #011/root/go/pkg/mod/google.golang.org/grpc@v1.29.1/server.go:1409 +0xc5c
Oct 22 00:53:44 bedb-master1 weed[8055]: google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0001ce8b0, 0xc0001f31e0, 0x23bb800, 0xc000ac5500, 0xc000ab7100)
Oct 22 00:53:44 bedb-master1 weed[8055]: #011/root/go/pkg/mod/google.golang.org/grpc@v1.29.1/server.go:746 +0xa5
Oct 22 00:53:44 bedb-master1 weed[8055]: created by google.golang.org/grpc.(*Server).serveStreams.func1
Oct 22 00:53:44 bedb-master1 weed[8055]: #011/root/go/pkg/mod/google.golang.org/grpc@v1.29.1/server.go:744 +0xa5
Oct 22 00:53:44 bedb-master1 systemd[1]: weedmaster.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Oct 22 00:53:44 bedb-master1 systemd[1]: weedmaster.service: Failed with result 'exit-code'.
 2020-10-21 23:15:48 -07:00
Chris Lu
2bc05cfbc8 fix typo 2020-10-21 22:26:41 -07:00
Chris Lu
4ff2ceee33 UI fix on rendering EC volumes 
addressing UI problem with https://github.com/chrislusf/seaweedfs/issues/1551
 2020-10-21 22:05:58 -07:00
Chris Lu
f5ee19e0db shortcut for appending operations 2020-10-21 20:44:01 -07:00
Chris Lu
b5917bf846 clean up 2020-10-21 19:31:26 -07:00
Chris Lu
9884dfc369 use bytebuffer for interval list 2020-10-21 19:29:51 -07:00
Chris Lu
bd103c143a add lock for vidCache 2020-10-21 19:28:59 -07:00
Konstantin Lebedev
05b5f12f2e multiplate DomainNames through comma 2020-10-21 20:48:51 +05:00
Chris Lu
81cf8d04df minor 2020-10-21 02:42:23 -07:00
Chris Lu
d90e7ee3cc adjust log level 2020-10-21 02:17:40 -07:00
Chris Lu
3bf0116de1 mount: less channel waiting 2020-10-21 02:16:21 -07:00
Chris Lu
c31b254248 mount: shortcut when there is only one chunk 2020-10-20 23:48:29 -07:00
Chris Lu
f64252023e Revert "a better byte buffer pool" 
This reverts commit 59c6422777.
 2020-10-20 23:21:54 -07:00
Chris Lu
ad652ffb97 avoid close closed channel 2020-10-20 22:54:34 -07:00
Chris Lu
20442dc1bc avoid extra data copying 2020-10-20 22:54:21 -07:00
Chris Lu
59c6422777 a better byte buffer pool 2020-10-20 22:53:58 -07:00
Chris Lu
6c92fa349d filer: support default rack 
fix https://github.com/chrislusf/seaweedfs/issues/1546
 2020-10-20 17:41:39 -07:00
Chris Lu
b3aa2fab9a s3: report error to s3 when updating an object but it is already a directory 
fix https://github.com/chrislusf/seaweedfs/issues/1545
 2020-10-20 10:25:16 -07:00
Chris Lu
410b818aa7 master: avoid timer leakage 2020-10-19 14:24:57 -07:00
Chris Lu
69f0da5d73 mount: pass along replication and collection parameters 2020-10-19 00:12:42 -07:00
Chris Lu
c0ab458671 report nil instead of EOF for empty files 
related to https://github.com/chrislusf/seaweedfs/issues/1541
 2020-10-17 11:03:46 -07:00
Chris Lu
898db14729 2.05 2020-10-17 02:42:33 -07:00
Chris Lu
09bab17aff mount: avoid "send on closed channel" 2020-10-16 12:54:23 -07:00
Chris Lu
9d80a3428c add debug_webdav 2020-10-16 12:53:56 -07:00
Chris Lu
37f165d743 webdav: return io.EOF when at end of a file 
fix https://github.com/chrislusf/seaweedfs/issues/1344
 2020-10-16 12:53:37 -07:00
Chris Lu
68d39c86f1 mysql, postgres, cassandra: change kv key to base64 encoding 
The exisitng key-value operation for stores using mysql, postgres, and maybe cassandra are already broken.

The kv is used to store hardlink, filer store signature and replication progress.

So users using hardlink and also uses mysql, postgres, or cassandra will have broken hard links.

Users using filer.sync will need to re-sync the files.
 2020-10-16 11:10:12 -07:00
Chris Lu
ee1fc6558a refactor 2020-10-16 11:02:44 -07:00
Chris Lu
06e6341097 set initial write time to avoid race condition with delayed chunks 2020-10-15 11:08:45 -07:00
Chris Lu
d598c47136 log errors 2020-10-15 11:01:43 -07:00
Chris Lu
5039aff310 release the goroutine to read errors 2020-10-15 10:59:18 -07:00
Chris Lu
ace0ea3d28 s3: avoid duplicated bucket 2020-10-15 10:52:20 -07:00
Konstantin Lebedev
dc2e13092d add number of read only volumes metric 2020-10-15 15:48:40 +05:00
Chris Lu
e714c28a02 mount: async write file chunk 2020-10-14 23:28:03 -07:00
Chris Lu
c95d3d9dff adjust tests 2020-10-14 12:27:52 -07:00
Chris Lu
93bcf56514 file read report EOF 
fix https://github.com/chrislusf/seaweedfs/issues/1344
 2020-10-14 12:18:24 -07:00
Chris Lu
1069b325dd shell: volumeServer.evacuate adds printout for ec volumes 2020-10-13 20:26:03 -07:00
Chris Lu
0542911e29 go fmt 2020-10-13 19:50:46 -07:00
Chris Lu
58fa506491 minor 2020-10-13 19:50:22 -07:00
Chris Lu
28d4e1a51b mount: retry for directory listing with filer 
related to https://github.com/chrislusf/seaweedfs/issues/1530
 2020-10-13 19:49:52 -07:00
Chris Lu
c127da1219 filer: linearize timeout for large chunk of data 2020-10-13 14:04:46 -07:00
Chris Lu
aac4cb1f0c adds errror on read and write 2020-10-13 13:53:34 -07:00
Chris Lu
9b4f7fed14 mount: report filer IO error 
related to https://github.com/chrislusf/seaweedfs/issues/1530
 2020-10-13 11:21:13 -07:00
Chris Lu
3f7d1d1bf1 Only wait on retryable requests 2020-10-13 00:29:46 -07:00
Chris Lu
b18f21cce1 mount: fix bound tree with filer.path 
fix https://github.com/chrislusf/seaweedfs/issues/1528
 2020-10-12 21:58:37 -07:00
Chris Lu
f022aff289 add back http.StatusPartialContent 
revert e7c04af1d0
 2020-10-12 12:26:25 -07:00
Chris Lu
bbd0afd37e 2.04 2020-10-11 21:25:30 -07:00
Chris Lu
7704469d29 mount: outsideContainerClusterMode changed to use volume server publicUrl 2020-10-11 20:42:15 -07:00
Chris Lu
723ae11db4 refactoring in order to adjust volume server url later 2020-10-11 20:15:10 -07:00
Chris Lu
d155f907c2 mount: configurable read wait time 2020-10-10 20:09:43 -07:00
Chris Lu
9b0e8ef026 filer: added QR code to transfer files with mobile devices 2020-10-10 18:00:21 -07:00
Chris Lu
8a52379ecb add retry if volume can not be found 2020-10-10 16:02:39 -07:00
Chris Lu
70af0ec24c filer: fix hanging on read 2020-10-10 16:02:10 -07:00
Chris Lu
cff8bb6554 return proper error 2020-10-10 15:43:22 -07:00
Chris Lu
6da8eef54f fix logic error 2020-10-09 00:01:47 -07:00
Chris Lu
5d01dd28a6 add logging 2020-10-08 23:38:00 -07:00
Chris Lu
8d34eb0050 mount:exponentially backoff if read error for about 10 minutes 2020-10-08 23:31:26 -07:00
Chris Lu
6e1f936efd refactoring 2020-10-08 23:19:42 -07:00
Chris Lu
b2ee5873fb fix error not being returned 2020-10-08 23:19:20 -07:00
Chris Lu
b067853162 mount: ignore recursion error 
fix https://github.com/chrislusf/seaweedfs/issues/1514

the host OS should manage recursive deletion.
 2020-10-08 21:55:55 -07:00
Chris Lu
e1a8e3f900 fix logging 2020-10-08 18:40:38 -07:00
Chris Lu
8676db0f6a adjust logging 2020-10-08 18:33:37 -07:00
Chris Lu
bd8b9b0c2e some changes to prepare fix after failed reads 2020-10-08 18:33:06 -07:00
Chris Lu
78859e35fc s3: adjust config examples 2020-10-08 10:12:09 -07:00
Chris Lu
9e7a2772b1 s3: only admin can list all buckets 2020-10-08 10:12:09 -07:00
Chris Lu
eed492b73b randomize file locations 2020-10-07 23:58:32 -07:00
Chris Lu
ec08a1670b volume: avoid hanging when stopping a volume server 2020-10-07 23:48:24 -07:00
Chris Lu
62d3d3aea0 go fmt 2020-10-07 23:30:54 -07:00
Chris Lu
e91b9c85a2 s3: Added support for "List" action in weed s3 -config=... in the config file. 
fix https://github.com/chrislusf/seaweedfs/issues/1511
 2020-10-07 23:22:35 -07:00
Chris Lu
a8624c2e4f read from alternative replica 
related to https://github.com/chrislusf/seaweedfs/issues/1512
 2020-10-07 22:49:04 -07:00
Chris Lu
da4edf3651 master: check peers for existing leader before starting a leader election 
fix https://github.com/chrislusf/seaweedfs/issues/1509
 2020-10-07 01:25:39 -07:00
Chris Lu
c543762e23 volume: auto expands -max setting 
I have tried to run weed volume but got errors...
cmd:
weed volume -max=0 -mserver="127.0.0.1:9333" -port=8080 -dir="/var/www/d1/sfs,/var/www/d2/sfs,/var/www/d3/sfs"
log:
I1006 15:09:26 17040 file_util.go:23] Folder /var/www/d1/sfs Permission: -rwxr-xr-x
I1006 15:09:26 17040 file_util.go:23] Folder /var/www/d2/sfs Permission: -rwxr-xr-x
I1006 15:09:26 17040 file_util.go:23] Folder /var/www/d3/sfs Permission: -rwxr-xr-x
F1006 15:09:26 17040 volume.go:142] 3 directories by -dir, but only 1 max is set by -max
What am I doing wrong? Thx.
 2020-10-06 09:05:30 -07:00
Chris Lu
36492c47ec adjust 2020-10-05 14:06:18 -07:00
Chris Lu
8f8e9ddbde adjust API 2020-10-05 09:47:07 -07:00
Konstantin Lebedev
e4f2d9eb4a We return etag using the same algorithm as aws s3 
https://teppen.io/2018/06/23/aws_s3_etags/
 2020-10-05 14:43:32 +05:00
Chris Lu
4fc673341f 2.03 2020-10-04 21:58:22 -07:00
Chris Lu
6b591b02af adjust log level 2020-10-04 16:21:43 -07:00
Устюжанин Антон Александрович
702b1cb876 fix: remove deleted peers if resumeState = true 2020-10-04 21:56:17 +05:00
Chris Lu
d522df0ecd adjust log level 2020-10-04 01:35:32 -07:00
Chris Lu
666859f7cd mount: read prefetching 2020-10-04 01:31:04 -07:00
Chris Lu
bc47835997 mount: avoid duplicated reads 2020-10-03 20:16:42 -07:00
Chris Lu
8d65ad1444 Revert "mount: adds read prefetching" 
This reverts commit 692f0614d1.
 2020-10-03 19:40:56 -07:00
Chris Lu
692f0614d1 mount: adds read prefetching 2020-10-03 16:50:09 -07:00
Chris Lu
707936f482 re-enable caching larger than 16MB 
revert 62ce85610e
 2020-10-03 14:12:38 -07:00
Chris Lu
2c20ef72ae fix starting problem 2020-10-03 13:56:38 -07:00
Chris Lu
d77e3c96e9 FUSE: use the mount directory permissions 2020-10-03 13:37:35 -07:00
Chris Lu
95ffed87c0
Merge pull request #1505 from ustuzhanin/remove_raft_state 
Resume raft state
 2020-10-03 10:38:47 -07:00
Устюжанин Антон Александрович
6fbfe73683 fix: resumeState 2020-10-03 22:16:47 +05:00
Устюжанин Антон Александрович
dc31b19469 fix: restore raft state 2020-10-03 14:03:41 +05:00
Chris Lu
f781cce500 s3: support object tagging 
* GetObjectTagging
* PutObjectTagging
* DeleteObjectTagging
 2020-10-02 22:21:51 -07:00
Устюжанин Антон Александрович
8c82fb7e5f fix: restore raft state 2020-10-02 23:01:20 +05:00
Chris Lu
9ab98fa912 s3 metrics adjust the label 2020-10-01 07:17:47 -07:00
Chris Lu
f1e879fe7a 2.02 2020-10-01 07:15:49 -07:00
Chris Lu
a1c01d716b volume: avoid deadlock when deleting volumes 
fix https://github.com/chrislusf/seaweedfs/issues/1501
 2020-10-01 07:10:03 -07:00
Konstantin Lebedev
86329bbf2b label name is statusCode 2020-10-01 01:22:38 +05:00
Konstantin Lebedev
68463e92c1 add status code in S3RequestCounter 2020-10-01 00:59:39 +05:00
Chris Lu
79ab10e300 adjust help message 2020-09-30 09:32:00 -07:00
Chris Lu
55cb68c09c filer: replication follows master config if not specified 2020-09-30 09:15:55 -07:00
Chris Lu
b9887504e8 fix test 2020-09-27 23:19:50 -07:00
Chris Lu
f46eae284e adjust for test 2020-09-27 23:08:11 -07:00
Chris Lu
75aca5e13d 2.01 2020-09-27 23:02:41 -07:00
Chris Lu
e6552b5e1e filer: able to start s3 together 2020-09-27 23:00:43 -07:00
Chris Lu
474e2b6ac3 add a hard link marker to 16byte + maker, for future extensions 2020-09-27 22:38:30 -07:00
Chris Lu
c49e2bb9a3 adjust 2020-09-27 12:07:45 -07:00
Chris Lu
62ce85610e skip caching too large chunks 2020-09-27 11:58:48 -07:00
Chris Lu
9ad2dcca2b more tests 2020-09-27 11:42:51 -07:00
Chris Lu
e43d86c796 fix pre allocated volume size 2020-09-27 10:58:19 -07:00
Chris Lu
31fc7bb2e1 refactor 
adjust for faster test
 2020-09-27 10:41:29 -07:00
Chris Lu
a37535cd9f avoid non utf-8 in filename 
fix https://github.com/chrislusf/seaweedfs/issues/1493
 2020-09-26 00:13:39 -07:00
Chris Lu
41be7a4c8f filer: upload also set file size 
ensure works same as fuse mount
 2020-09-25 12:02:06 -07:00
Chris Lu
35f6518c36 weed upload: usePublicUrl was not being used on big files 
fix https://github.com/chrislusf/seaweedfs/issues/1492
 2020-09-25 02:41:38 -07:00
Chris Lu
22049dc2e6
Merge pull request #1491 from hilimd/master 
s3:  fix get object metadata problem
 2020-09-25 02:35:00 -07:00
limd
402aef8f30 s3: 
1.fix spark reading S3 directory wildcard problem
2.fix the problem of the spark history service writing S3 directory

reference git revsion number: b41b7ea4d0
 2020-09-25 14:37:02 +08:00
Chris Lu
8516517c48 filer store: Cassandra supports username/password 2020-09-24 21:31:06 -07:00
limd
ed7816681a Add cassandra authenticator mode 2020-09-25 11:22:47 +08:00
limd
370a98cf6b code style 2020-09-25 11:19:42 +08:00
limd
aee3fd08b9 code style 2020-09-25 11:04:32 +08:00
Chris Lu
043b063136
Merge pull request #1482 from hilimd/master 
Fix: s3 delete object
 2020-09-24 18:21:34 -07:00
limd
48c578410f mount: rollback default value 2020-09-25 09:18:52 +08:00
Chris Lu
dbf5327b98 s3: handle response-content-encoding 
fix https://github.com/chrislusf/seaweedfs/issues/1487
 2020-09-24 18:09:52 -07:00
Chris Lu
7726965a47 minor 2020-09-24 11:20:12 -07:00
Chris Lu
0790c6d605 fix empty hard link id 2020-09-24 11:16:43 -07:00
Chris Lu
1012df7bb5 switch hardlink id from int64 to bytes 2020-09-24 11:11:42 -07:00
Chris Lu
4856bce0ee adjust for metrics port 2020-09-24 10:21:23 -07:00
Chris Lu
d2d3aec3e1 consolidate to one metricsPort in "weed server" mode 2020-09-24 09:55:02 -07:00
Chris Lu
2e9099369e
Merge pull request #1489 from kmlebedev/promhttp 
Promhttp
 2020-09-24 09:50:48 -07:00
Chris Lu
2e7c361a0d hardlink deletion factors in hardlink counter 2020-09-24 09:43:52 -07:00
Chris Lu
1295347958 adjust hardlink update 
simplify logic, pass entity content directly to hard link. The "weed mount" handles the logic to calculate hard link counter.
 2020-09-24 09:43:00 -07:00
Konstantin Lebedev
98e9de6e11 fix style 2020-09-24 17:48:39 +05:00
Konstantin Lebedev
324e44d4b3 add start metrics server 2020-09-24 17:45:39 +05:00
limd
cbca14edc5 mount: fix k8s pvc and os mount directory permission bug 2020-09-24 18:07:16 +08:00
Chris Lu
5e239afdfc hardlink works now 2020-09-24 03:06:48 -07:00
limd
59e91e9c7e mount: fix k8s pvc mount directory permission 2020-09-24 10:25:45 +08:00
Chris Lu
c7d7b1a0f6
Merge pull request #1485 from LIBA-S/fix_oversized 
Correct the oversized state of volume after compaction
 2020-09-23 19:24:30 -07:00
Chris Lu
b80a2b3cc9
Merge pull request #1486 from LIBA-S/fix_race_condition 
Fix a race condition when handle VolumeLocationList
 2020-09-23 19:24:16 -07:00
LIBA-S
eecd6b5d35 Fix a race condition when handle VolumeLocationList 2020-09-23 20:56:51 +08:00
LIBA-S
0157798ebf Correct the oversized state of volume after compaction 2020-09-23 20:27:42 +08:00
limd
9b21ec27e4 mount: auto created dir set corrct umask fix bug 2020-09-23 18:35:37 +08:00
hilimd
0db149fb5f
Merge pull request #18 from chrislusf/master 
sync
 2020-09-23 17:32:05 +08:00
Chris Lu
f7a0ccb595 mount: auto created dir set corrct umask 2020-09-23 02:31:19 -07:00
Chris Lu
f0e325b6fc mount: auto created directory follow umask 2020-09-23 02:27:57 -07:00
limd
d506080c36 rollback 2020-09-23 14:29:53 +08:00
limd
8f9f29b773 fixed the problem of empty directory when S3 deleted the directory 2020-09-23 13:33:13 +08:00
limd
1892677b22 fixed the problem of empty directory when S3 deleted the directory 2020-09-23 13:15:06 +08:00
hilimd
85570f27a0
Merge pull request #17 from chrislusf/master 
sync
 2020-09-23 11:39:57 +08:00
limd
a99f63cb4d 1.add S3 copy directory function 
2.fixed the problem of empty directory when S3 deleted the directory
 2020-09-23 11:26:01 +08:00
Chris Lu
b61d33f251 expose only store wrapper to meta changes 2020-09-22 16:27:36 -07:00
Chris Lu
0adbb56cc1 rename 2020-09-22 16:24:13 -07:00
Chris Lu
3e52329cee Revert "Merge pull request #1479 from LIBA-S/fix_oversized" 
This reverts commit bd11f0b3e4, reversing
changes made to ec5b9f1e91.
 2020-09-22 15:05:37 -07:00
Chris Lu
4a1fe4b8e2 add error logs 2020-09-22 09:16:10 -07:00
LIBA-S
8573ac82c6 Fix: remove the oversized state after compaction 2020-09-22 21:48:39 +08:00
James Hartig
658fc2e5b6 Allow option to enable volume pprof on server 2020-09-21 22:43:10 -04:00
James Hartig
91e4eca1e9 Fix deadlock with KeepConnected and SendHeartbeat 
There's the potential where we're writing to a clientConn and it goes away
and we're stuck keeping a read lock on clientChansLock. This causes
KeepConnected to not be able to remove the client since it requires a write
lock on clientChansLock. This ends up backing up SendHeartbeat because it
can't get a read lock.
 2020-09-21 22:41:38 -04:00
Chris Lu
9cdbfc1a49 refactor 2020-09-21 17:34:38 -07:00
Chris Lu
63373a9f9f filer: add file extended properties 2020-09-21 11:08:34 -07:00
Chris Lu
b9c1f3e9de s3: fixes for list multipart upload 2020-09-21 10:51:24 -07:00
Chris Lu
289e62a305 master: better locking of in memory volume data 
related to https://github.com/chrislusf/seaweedfs/issues/1436#issuecomment-695880135
 2020-09-20 23:07:55 -07:00
Chris Lu
9a3b564508 adjust printout 2020-09-20 16:01:56 -07:00
Chris Lu
62563a895a refactoring 2020-09-20 16:00:01 -07:00
Chris Lu
c2faab23b6 refactor 2020-09-20 15:40:49 -07:00
Chris Lu
d013d09a9b adjust logging 2020-09-20 15:38:59 -07:00
Chris Lu
f498c71199 shell: move volume operations to use flag parsing arguments 2020-09-20 09:27:34 -07:00
Chris Lu
fcbc520373 fix print out 2020-09-20 08:47:30 -07:00
Chris Lu
56094541c8 shell: adjust command option for collection.delete 2020-09-20 08:46:16 -07:00
Chris Lu
9cac9c4001 2.00 2020-09-20 00:47:04 -07:00
Chris Lu
dc4d2145ff open up 2020-09-19 20:23:12 -07:00
Chris Lu
29abe980df s3: add support for PostPolicy 
fix https://github.com/chrislusf/seaweedfs/issues/1426
 2020-09-19 20:14:19 -07:00
Chris Lu
41d508edfd go fmt 2020-09-19 14:10:26 -07:00
Chris Lu
5b40a2690a refactoring 2020-09-19 14:09:58 -07:00
Chris Lu
2c21eb1971 volume: get metrics configuration from master 
fix https://github.com/chrislusf/seaweedfs/issues/1354
 2020-09-19 00:03:00 -07:00
Chris Lu
2cbd1cf121 fix compilation 2020-09-18 00:15:54 -07:00
Chris Lu
23e9ede068 s3: collect metrics 2020-09-18 00:09:04 -07:00
Chris Lu
852e5f7cbc filer: fix mongodb insert 
fix https://github.com/chrislusf/seaweedfs/issues/1471
 2020-09-17 21:50:52 -07:00
Chris Lu
6544e60bea s3 add metrics 
empty for now
 2020-09-17 06:56:15 -07:00
Chris Lu
cb427d48fa filer report metrics configuration 2020-09-17 06:46:51 -07:00
Chris Lu
e4e0234998 refactoring 2020-09-17 06:43:54 -07:00
Chris Lu
feca07bf96 Filer: ip bind 
fix https://github.com/chrislusf/seaweedfs/issues/1470
 2020-09-16 10:26:08 -07:00
Chris Lu
e861a6a3ab simplify metrics configuration logic 2020-09-16 01:39:30 -07:00
Chris Lu
be54eeb364 passing value by checkWithMaster() instead 2020-09-16 01:33:45 -07:00
Chris Lu
4d21de63ee go fmt 2020-09-16 01:27:24 -07:00
Chris Lu
913a16268d volume: load configuration from master at the start 
fix https://github.com/chrislusf/seaweedfs/issues/1469
 2020-09-16 01:27:05 -07:00
Chris Lu
c9202c4b3d add storage backend to GetMasterConfigurationResponse 2020-09-16 01:12:22 -07:00
Chris Lu
8a0710cb73 handle more than 2GB files 
related to https://github.com/chrislusf/seaweedfs/issues/1468
 2020-09-16 00:41:50 -07:00
Chris Lu
854007bc98 minor 2020-09-16 00:37:57 -07:00
Chris Lu
10f9081526 filer: adjust meta data events to received timestamp 
if a client is already connected and start from t0. A message recieved at t+1 but with timestamp t-1 may not be processed by the client.

This commit changes to the event received time, so the replication can be ordered.
 2020-09-15 01:18:33 -07:00
Chris Lu
684a875876 update help message 2020-09-15 00:40:38 -07:00
Chris Lu
2d3a904a82 shell: volumeServer.evacuate adds option to skip non moveable volumes 2020-09-15 00:33:49 -07:00
Chris Lu
a595916342 shell: add volumeServer.evacuate command 2020-09-14 23:47:11 -07:00
Chris Lu
103fafe00b weed export: print out [start,stop) content range 2020-09-14 22:57:23 -07:00
Chris Lu
ada996fe55 filer: support createing empty folder 
fix https://github.com/chrislusf/seaweedfs/issues/1161
 2020-09-14 13:07:46 -07:00
Chris Lu
200fe5c83e go fmt 2020-09-13 21:26:30 -07:00
Chris Lu
5d6753fb98 shell: add volumeServer.leave command 2020-09-13 21:25:51 -07:00
Chris Lu
1af95c5b76 refactoring 2020-09-13 12:41:26 -07:00
Chris Lu
f8fea19669 1.99 2020-09-12 13:46:33 -07:00
Chris Lu
b0c7de186d filer: fix postgres prefixed directory listing problem 
fix https://github.com/chrislusf/seaweedfs/issues/1465
 2020-09-12 13:37:03 -07:00
Chris Lu
1a7afe7e6a adjust default value 
related to https://github.com/chrislusf/seaweedfs/issues/1453
 2020-09-12 12:46:42 -07:00
Chris Lu
f2723c1bc8 do not idx file format 
revert c9ab8d05fa
 2020-09-12 12:42:36 -07:00
Chris Lu
ba984a4e29 1.98 2020-09-12 04:13:02 -07:00
Chris Lu
446e476a11 go fmt 2020-09-12 04:08:03 -07:00
Chris Lu
ea26a98753 volume: validate volume correctness if last entry is a deletion 2020-09-12 04:07:04 -07:00
Chris Lu
d15682b4a1 shell: volume.balance plan by ratio of fullness 2020-09-12 04:06:26 -07:00
Chris Lu
c0ee78d2fa adjust make file 2020-09-12 04:05:42 -07:00
Chris Lu
cd9b89ba55 reduce default wait time to 10s before shutting down 2020-09-12 04:05:33 -07:00
Chris Lu
e2c741f76f adjust replica placement after move 2020-09-12 01:01:19 -07:00
Chris Lu
2a0925590c filer: etcd store avoid read with nil option 
fix https://github.com/chrislusf/seaweedfs/issues/1463
 2020-09-11 15:29:45 -07:00
Chris Lu
3984c3962f add comment 2020-09-11 15:07:19 -07:00
Chris Lu
baa6bdf4d4 s3: listMultipartUploads fix output format 2020-09-11 15:04:01 -07:00
Chris Lu
3eda8d6dfc s3: ListParts output xml format 
fix https://github.com/chrislusf/seaweedfs/issues/1461
 2020-09-11 14:53:50 -07:00
Chris Lu
ab201c2798 1.97 2020-09-11 13:47:50 -07:00
Chris Lu
9d4bdfcfdf fix volume integrity checking 2020-09-11 11:34:10 -07:00
Chris Lu
1b8094ef75 weed export: export deleted files 2020-09-11 02:05:14 -07:00
Chris Lu
12a8f5294d test for multi dc replication 2020-09-11 01:21:17 -07:00
Chris Lu
e60b2117c3 shell: volume balance follows replica placement 2020-09-11 00:29:25 -07:00
Chris Lu
89a62e8007 refactoring 2020-09-10 23:05:00 -07:00
Chris Lu
19537c9d21 1.96 2020-09-10 19:52:07 -07:00
Chris Lu
eaf9fdde99 avoid sharing context over separate goroutine 2020-09-10 19:51:03 -07:00
Chris Lu
cc5fe6f5ee handle special characters in html link 2020-09-10 19:46:02 -07:00
Chris Lu
401ccf1509 fix test 2020-09-10 15:13:23 -07:00
Chris Lu
c9ab8d05fa fixes for reading deleted fid 2020-09-10 14:42:52 -07:00
ruitao.liu
5b0676049a change elastic initialize process similar as others. 2020-09-10 23:35:20 +08:00
Chris Lu
660d7c0edd 1.95 2020-09-10 01:32:05 -07:00
Chris Lu
0d9b858cfb
Merge pull request #1455 from taozix/master 
add more basic elastic options.
 2020-09-10 01:31:17 -07:00
ruitao.liu
9be4e97625 change logs print format. 2020-09-10 16:30:15 +08:00
ruitao.liu
719dc43af1 modify elastic urls from string to array. 2020-09-10 16:24:09 +08:00
ruitao.liu
72f9d7f047 use util to generate md5. 2020-09-10 16:11:18 +08:00
ruitao.liu
6a5b38c0d4 fix elastic kv ops. 2020-09-10 15:59:16 +08:00
ruitao.liu
3f7fbfddca add more basic elastic options. 2020-09-10 14:22:07 +08:00
Chris Lu
b183ae54b5 master: changing 301 to 308 on redirect to volume servers 
fix https://github.com/chrislusf/seaweedfs/issues/1454
 2020-09-09 22:36:10 -07:00
Chris Lu
47b3f932e4 watch: adjust output format 2020-09-09 22:34:48 -07:00
Chris Lu
daf0a449f7 properly cancel context for streaming grpc 2020-09-09 12:07:15 -07:00
Chris Lu
7f69acd1f2 sync pprof 2020-09-09 11:33:52 -07:00
Chris Lu
387ab6796f filer: cross cluster synchronization 2020-09-09 11:21:23 -07:00
Chris Lu
4fc0bd1a81 return http response directly 2020-09-09 03:53:09 -07:00
Chris Lu
5e13bc878c adjust log level 2020-09-09 03:49:26 -07:00
Chris Lu
d1b816212f return nil if no need to do anything 2020-09-07 16:03:05 -07:00
Chris Lu
64a621bcc8 shell: volume.fix.replication also purge over replicated volumes 2020-09-07 16:00:10 -07:00
Chris Lu
c18ea21f7a 1.94 2020-09-07 13:13:53 -07:00
Chris Lu
18b98fdb72 fix test 2020-09-07 12:57:38 -07:00
Chris Lu
44b3f2efc8 filer: Elastic Search return correct kv error 
fix https://github.com/chrislusf/seaweedfs/issues/1452
 2020-09-07 12:44:51 -07:00