mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-12-25 10:17:55 +08:00
Code Issues Packages Projects Releases Wiki Activity
5,735 Commits 32 Branches 296 Tags 153 MiB
302738ee74
Commit Graph

3679 Commits

Author SHA1 Message Date
Chris Lu
ca998328c2 do not add new volumes when below minFreeSpacePercent 
fix https://github.com/chrislusf/seaweedfs/issues/2017
 2021-04-21 23:11:11 -07:00
Chris Lu
cd7bf1a72a filer.copy copy empty folders 
fix https://github.com/chrislusf/seaweedfs/issues/2016
 2021-04-21 11:17:43 -07:00
Chris Lu
a8114da02d avoid thundering herd effect 
transient errors may cause thundering herd effect to all trying to recover from remove ec shards
 2021-04-21 10:17:12 -07:00
Chris Lu
ae74d8f02a fix error message 
related to https://github.com/chrislusf/seaweedfs/issues/2012
 2021-04-21 01:40:16 -07:00
Chris Lu
11c405fc85 ensure file handles are released 2021-04-20 19:56:55 -07:00
liuxiaobo
c31c5e829c
fix path-specific filer store comment error 2021-04-20 10:08:58 +08:00
Chris Lu
83cf94ad2d delay new file creation unless file is opened exclusively 2021-04-19 10:58:25 -07:00
Chris Lu
e983f91b03 2.40 2021-04-18 13:58:01 -07:00
Chris Lu
d1c813c470 let the fuse library manage directory id 
otherwise, on mac, during large directory deletion, if some ReaDirAll happens, the lib seems confused about the directories, and some child directories are not deleted.
 2021-04-18 13:07:28 -07:00
Chris Lu
372872ebbf set root node inode number 2021-04-18 13:07:28 -07:00
Chris Lu
e332da4837 set inode value 2021-04-18 13:07:28 -07:00
Chris Lu
2acf6be24e resend the http request if connection is stale 2021-04-18 13:07:28 -07:00
Chris Lu
6cbd786db9 correctly runs git clone 2021-04-18 13:07:28 -07:00
Chris Lu
d41e6826d3 adjust logging 2021-04-18 13:06:38 -07:00
Chris Lu
d9a2a7f1c4 WIP 
no memory issue

if some directory is removed, it may have this error

$ rm -Rf ~/tmp/m2/s1
rm: fts_read: Device not configured
 2021-04-18 13:06:38 -07:00
Chris Lu
54410ca955 cleaner way to set readonly 2021-04-18 10:02:02 -07:00
Konstantin Lebedev
198688c717 revert volume etag 2021-04-16 23:22:31 +05:00
Chris Lu
c83ab91e2e remove unused variable 2021-04-16 10:34:02 -07:00
Chris Lu
3074e9b428 ensure consistent inode value 2021-04-15 22:42:24 -07:00
Chris Lu
b971317a16 avoid possible corrupted file names 2021-04-15 11:41:34 -07:00
Chris Lu
283d703d50 adjust text 2021-04-15 11:29:58 -07:00
Chris Lu
609e228578 avoid forward slash in file names 2021-04-15 10:53:04 -07:00
Chris Lu
ba92f2e714 add node.selectedVolumes 
fix https://github.com/chrislusf/seaweedfs/issues/1990
 2021-04-15 03:19:28 -07:00
Chris Lu
16c0304416 ensure to delete on filer also 2021-04-15 02:29:04 -07:00
Chris Lu
217e0f9066 mount: remove folder recursively 2021-04-15 01:51:10 -07:00
Chris Lu
3e669e6d7b mostly refactoring, add some error handling 2021-04-14 23:33:37 -07:00
Chris Lu
07f712c83f fix typo 2021-04-14 23:21:38 -07:00
Chris Lu
36c79de3f4 fuse mount: dir ReadDirAll avoid extra conversion to filer_pb.Entry 2021-04-14 23:21:24 -07:00
Chris Lu
e41766feb6 fuse mount: dir lookup avoids extra conversion to filer_pb.Entry object 2021-04-14 22:38:34 -07:00
Chris Lu
1adc8f86ea lighten up File object 
file.entry only exists when file.isOpen
 2021-04-14 20:49:15 -07:00
Chris Lu
6bc09b18c4 truncate is a bit faster to reuse the storage 2021-04-14 20:26:56 -07:00
Chris Lu
c04b7e106f mount: remove entry from Dir object 2021-04-14 20:26:13 -07:00
Chris Lu
fc0cbf565f add option to obfuscate the file names 2021-04-14 15:37:24 -07:00
Chris Lu
e75633c64f volume.check.disk: break loop for read only volumes 
fix https://github.com/chrislusf/seaweedfs/issues/2002
 2021-04-14 12:40:13 -07:00
Chris Lu
9d50867d08 volume.tier.move: avoid data loss when destination volume server already has the volume 
fix https://github.com/chrislusf/seaweedfs/issues/2001
 2021-04-14 10:26:26 -07:00
Chris Lu
ff4c1d5965 adjust logging 
fix https://github.com/chrislusf/seaweedfs/issues/1999
 2021-04-14 10:04:26 -07:00
qieqieplus
270645f8d7 fix #1996 2021-04-14 18:29:28 +08:00
Chris Lu
90677e1097 ensure to call line.Close() 
fix https://github.com/chrislusf/seaweedfs/issues/1995

similar to https://github.com/peterh/liner/issues/104
 2021-04-14 01:33:21 -07:00
Chris Lu
3f3268cd1b go fmt 2021-04-14 00:30:16 -07:00
Chris Lu
ca0f07a188 move file reader, entryViewCache to file handle 
reduce file object size
 2021-04-14 00:29:58 -07:00
Chris Lu
5985a7d38d add log level possible values 
fix https://github.com/chrislusf/seaweedfs/issues/1989
 2021-04-12 21:15:51 -07:00
Chris Lu
f5de42fae3
Merge pull request #1975 from kmlebedev/iam_handlers 
IAM handlers
 2021-04-12 12:07:45 -07:00
Chris Lu
1e033d45b8 simpler logic 
related to https://github.com/chrislusf/seaweedfs/pull/1981
 2021-04-12 12:04:53 -07:00
Chris Lu
52200a903b
Merge pull request #1981 from ueni-ltd/if-modified-since 
Fix If-Modified-Since behavior
 2021-04-12 12:02:54 -07:00
Chris Lu
519b0e1e49 filer: upload to a directory without "/" suffix 
fix https://github.com/chrislusf/seaweedfs/issues/1988
 2021-04-12 11:56:56 -07:00
Konstantin Lebedev
8e02e138ea Merge branch 'upstreamMaster' into iam_handlers 2021-04-12 11:22:51 +05:00
Chris Lu
742ab1ec81 2.39 2021-04-11 19:47:11 -07:00
Chris Lu
0df5b53ad8 adjust help message 2021-04-11 00:26:28 -07:00
Chris Lu
f62c153274 go fmt 2021-04-10 23:48:18 -07:00
Chris Lu
af313dff58 add gateway for easier POST and DELETE blobs 2021-04-10 23:47:47 -07:00
Konstantin Lebedev
5021bea698 GetUserPolicy 2021-04-10 23:57:45 +05:00
Chris Lu
0b82edc0d2 filer: avoid stuck uploader 
fix https://github.com/chrislusf/seaweedfs/issues/1980

reverting the file upload batch executor
 2021-04-09 13:05:15 -07:00
Chris Lu
93f4146ffa properly release the view cache 2021-04-09 12:36:39 -07:00
Merlin Gaillard
f952f979d1 filer: return 304 when If-Modified-Since == Last-Modified 2021-04-09 15:04:17 +02:00
Merlin Gaillard
4d4acc715e s3api: handle 304 response code from filer 2021-04-09 12:13:19 +02:00
Chris Lu
6deb647a8f mount: fix possible memory leak 
if many files are read repeatedly, their metadata are accumulated in memory. This fix cleared the metadata after the file is read.
 2021-04-08 19:47:31 -07:00
Konstantin Lebedev
ba175f81b5 add auth aws signV4 2021-04-08 17:40:47 +05:00
Konstantin Lebedev
995ae91007 add DeleteUserPolicy 2021-04-08 11:16:36 +05:00
Philippe Pepiot
42a761ee20
Fix typo in weed filer long help 2021-04-07 22:47:23 +02:00
Chris Lu
3be061994f skip connection reset error 
fix https://github.com/chrislusf/seaweedfs/issues/1971

this is because the connections are pooled but the volume server has reset the connection
 2021-04-07 00:54:13 -07:00
Chris Lu
c5b08bac1b remove mac specific mount options 2021-04-07 00:54:13 -07:00
Konstantin Lebedev
f5f8eec8e2 fix get filerGrpcAddress 2021-04-06 13:53:56 +05:00
Konstantin Lebedev
011e6e90ee Merge branch 'upstreamMaster' into iamapipr 2021-04-06 13:50:33 +05:00
Konstantin Lebedev
ed79baa30f add tests 2021-04-06 13:43:08 +05:00
Chris Lu
2327c0756b fix to avoid loop 2021-04-05 23:24:26 -07:00
Chris Lu
a37eca78cd 2.38 2021-04-05 19:41:54 -07:00
Chris Lu
c131764c34 ensure error is sent back in channel 
fix https://github.com/chrislusf/seaweedfs/issues/1966

avoid shared readErr variable
 2021-04-05 19:40:12 -07:00
Chris Lu
0f64f5b9c8 mount: add readOnly option 
fix https://github.com/chrislusf/seaweedfs/issues/1961
 2021-04-04 21:40:58 -07:00
Chris Lu
6eee200c13 2.37 2021-04-04 18:45:48 -07:00
Chris Lu
fbb82a5c9c skip limiting if limit is zero 2021-04-04 18:38:33 -07:00
Chris Lu
8251d1140e refactor 2021-04-04 18:38:33 -07:00
Chris Lu
5dca98fe71 fix typo 2021-04-03 11:37:13 -07:00
Chris Lu
5e64f65632 ensure tables are created 
fix https://github.com/chrislusf/seaweedfs/issues/1957
 2021-04-02 12:02:26 -07:00
Chris Lu
319743d330 go fmt 2021-04-02 02:22:59 -07:00
Chris Lu
8e404a1433 go fmt 2021-04-02 02:22:26 -07:00
Chris Lu
bdf2ddddfd revert to same implementation as before 
This reverts commit 7e8edc3c4a.
 2021-04-02 02:21:38 -07:00
Chris Lu
67e019d54b add missing changes 2021-04-02 01:11:44 -07:00
Chris Lu
7e8edc3c4a refactoring 2021-04-02 01:10:24 -07:00
Chris Lu
cc0df36a9e smaller file chunks are faster 2021-04-01 02:21:59 -07:00
Chris Lu
cefe66f159 dedicated upload processor 
avoid thundering effect of overloading volume servers
 2021-04-01 02:21:40 -07:00
Chris Lu
1f984d2645 refactor buffer pool 2021-04-01 02:20:00 -07:00
Chris Lu
b5880334fc refactor 2021-03-30 21:07:34 -07:00
Chris Lu
e79e2ddeed remove unused MoveEvents 2021-03-30 20:42:44 -07:00
Chris Lu
e06807a55b adjust logs 2021-03-30 20:36:06 -07:00
Chris Lu
ac875976c0 filer, volume: add concurrent upload size limit to avoid OOM 
add some back pressure when writes are slow
 2021-03-30 02:10:53 -07:00
Chris Lu
a1e18a1384 server add memory profiling 2021-03-30 02:10:53 -07:00
LazyDBA247-Anyvision
9385e2224d fix postgres2 upsertQuery syntax also 2021-03-30 10:51:52 +03:00
Chris Lu
a5662c9e5e adjust ordering 2021-03-30 00:26:57 -07:00
Chris Lu
bec74c3e6a
Merge pull request #1948 from LazyDBA247-Anyvision/master 
add enableUpsert=true
 2021-03-30 00:25:01 -07:00
LazyDBA247-Anyvision
9f1cab179c use a compatible pg syntax for upsert 2021-03-30 10:21:27 +03:00
Chris Lu
a95929e53c reduce default concurrentWriters to 32 2021-03-30 00:17:52 -07:00
LazyDBA247-Anyvision
7f44d953b5 fix GetBool 2021-03-30 01:36:02 +03:00
LazyDBA247-Anyvision
4c51e6a660 add enableUpsert=true 
and rename config to upsertQuery
 2021-03-30 00:32:03 +03:00
Chris Lu
b61e9f6a1a adjust ordering 2021-03-29 00:49:50 -07:00
Chris Lu
1ebdbad222
Merge pull request #1946 from LazyDBA247-Anyvision/master 
Adding Customisation - insertQuery for postgres/mysql
 2021-03-29 00:42:48 -07:00
Konstantin Lebedev
8a95f9c10c iam GetUser 2021-03-29 12:01:44 +05:00
LazyDBA247-Anyvision
96c62bd34d scaffold - insertQuery 
Example for upsert query for mysql & postgres variants
 2021-03-29 10:01:12 +03:00
LazyDBA247-Anyvision
4a02389eb0 Adding custom insertQuery support for postgres/2 mysql/2 2021-03-29 09:58:13 +03:00
Chris Lu
6b7aa9633f 2.36 2021-03-28 19:09:06 -07:00
Chris Lu
bd7471d877 refactor 2021-03-25 12:05:59 -07:00
李海
06be5dc6c3 log snowflake sequencer nodeid's hex when start, it'll be in part of new assigned fid 2021-03-25 18:59:40 +08:00
李海
69b2dab9c6 add a snowflake sequencer as more robust fid generator, but less compressable than small auto-inc id 2021-03-25 18:49:26 +08:00
Konstantin Lebedev
40938d6a47 SaveInsideFiler S3 Configuration 2021-03-25 12:34:11 +05:00
Konstantin Lebedev
9f26f2815c SaveAs S3 Configuration 2021-03-25 12:34:11 +05:00
Konstantin Lebedev
82b0463fac handler PutUserPolicy 
https://docs.aws.amazon.com/IAM/latest/APIReference/API_PutUserPolicy.html
 2021-03-25 12:34:11 +05:00
Konstantin Lebedev
d7719d0542 base handlers 2021-03-25 12:34:11 +05:00
Konstantin Lebedev
03c7953254 init Iam Api Server 2021-03-25 12:34:11 +05:00
Chris Lu
82dfe06066
Merge pull request #1933 from kmlebedev/missingKeysThreshold 
add missingKeysThreshold for fsck
 2021-03-24 10:56:25 -07:00
Konstantin Lebedev
020a5d40c3 avoid counter is null 2021-03-24 22:51:22 +05:00
Konstantin Lebedev
df6cf0a2fa nonRepairThreshold 2021-03-24 22:07:13 +05:00
Konstantin Lebedev
9ce82a9cd2 GOMAXPROCS use defaults to the value of runtime.NumCPU 
https://github.com/chrislusf/seaweedfs/issues/1942
 2021-03-24 18:45:58 +05:00
Konstantin Lebedev
60972f1c97 rename option repairThreshold 2021-03-24 13:24:49 +05:00
Chris Lu
a801332b0d filer: return 409 if file conflicts with a directory 
fix https://github.com/chrislusf/seaweedfs/issues/1938
 2021-03-24 01:03:11 -07:00
Chris Lu
c42b95c596 more help message 
fix https://github.com/chrislusf/seaweedfs/issues/1937
 2021-03-23 17:27:57 -07:00
Konstantin Lebedev
40a9e88c07 rename var 2021-03-23 21:42:35 +05:00
Konstantin Lebedev
d21a5bf5d0 add missingKeysThreshold for fsck 2021-03-23 15:04:07 +05:00
Chris Lu
85cff10787 mysql: ensure name column is case sensitive 
fix https://github.com/chrislusf/seaweedfs/issues/1931
 2021-03-23 00:46:50 -07:00
Chris Lu
4abb511db3 make a local copy of the in memory cached data 2021-03-22 22:33:07 -07:00
Chris Lu
3cbc40fa48 avoid creating multiple reader 2021-03-22 22:32:47 -07:00
Chris Lu
2f7c7afdec refactor: remove unused parameter 2021-03-22 22:13:19 -07:00
Chris Lu
d48dd0c738 use []byte directly instead of bytes.Buffer 2021-03-22 22:12:57 -07:00
Chris Lu
8eb14967ef shell: volume.check.disk adds optional slow mode 
fix https://github.com/chrislusf/seaweedfs/issues/1930
 2021-03-22 21:01:43 -07:00
Chris Lu
288369cfc7 mount: release resources when Forget() is called 
address https://github.com/chrislusf/seaweedfs/issues/1929
 2021-03-22 16:22:19 -07:00
Chris Lu
5d931eff27 avoid possible nil 
fix https://github.com/chrislusf/seaweedfs/issues/1928

The nil was because of `dn.Parent().UnlinkChildNode(dn.Id())` in topo.UnRegisterDataNode() function, when the dn leaves the cluster.
 2021-03-22 13:24:07 -07:00
Chris Lu
f315eb2bb8 fix build error 2021-03-22 00:12:53 -07:00
Chris Lu
1dd5bc134c 2.35 2021-03-22 00:05:09 -07:00
Chris Lu
b465095db1 shell: add volume.check.disk to fix inconsistency for replicated volumes 
fix https://github.com/chrislusf/seaweedfs/issues/1923
 2021-03-22 00:03:16 -07:00
Chris Lu
df461402cc ensure entry attribute is not nill 
fix https://github.com/chrislusf/seaweedfs/issues/1926
 2021-03-21 22:27:30 -07:00
Chris Lu
102a951377 refactor, split into 2 files 2021-03-21 13:05:53 -07:00
Chris Lu
f6e2566ee5 better logs for cookie mismatch 2021-03-20 12:02:57 -07:00
Chris Lu
27e24162f2 minor 2021-03-20 06:37:03 -07:00
Chris Lu
41cc6e661b fs.mv ensures there are 2 arguments 2021-03-20 06:34:13 -07:00
Chris Lu
b2c1c209a5 refactor 2021-03-20 06:34:13 -07:00
Chris Lu
e9b1853d63 go fmt 2021-03-19 01:31:56 -07:00
Chris Lu
b1a86cf808 s3: copy object to itself 
fix https://github.com/chrislusf/seaweedfs/issues/1922
 2021-03-19 01:31:49 -07:00
Konstantin Lebedev
bf94149920 add line 2021-03-17 23:41:34 +05:00
Konstantin Lebedev
c5705e7a35 dir + slash key does not exist 
https://github.com/chrislusf/seaweedfs/issues/1917
 2021-03-17 23:40:42 +05:00
Konstantin Lebedev
eb54993a4e Merge branch 'upstreamMaster' into check_chunkviews_mr 
# Conflicts:
#	weed/filer/filechunk_manifest.go
#	weed/filer/stream.go
#	weed/replication/repl_util/replication_util.go
#	weed/util/fasthttp_util.go
 2021-03-16 15:29:49 +05:00
Chris Lu
9672f9e1b2 2.34 2021-03-16 03:01:15 -07:00
Chris Lu
69694a17be reverting 7d57664c2d 2021-03-16 02:59:26 -07:00
Chris Lu
c00dd5e62e report error if entry not found 2021-03-16 02:55:49 -07:00
Konstantin Lebedev
06da02739d CheckAllChunkViews() for HEAD requests only 2021-03-16 14:15:17 +05:00
Chris Lu
91a3ac9731 2.33 2021-03-16 00:36:06 -07:00
Chris Lu
4b1ed227d1 revert fasthttp changes 
related to https://github.com/chrislusf/seaweedfs/issues/1907
 2021-03-16 00:33:14 -07:00
Chris Lu
10164d0386
Merge pull request #1905 from wuh-fnst/s3ListCounterErr 
s3: count correction for the number of files in the dir
 2021-03-15 20:54:58 -07:00
Chris Lu
22a07a12d2 [mount] [regression] filer.path option results in empty mount in 2.32 
fix https://github.com/chrislusf/seaweedfs/issues/1906
 2021-03-15 20:46:25 -07:00
wuh-fnst
050a7f9599 s3: count correction for the number of files in the dir 2021-03-16 09:22:03 +08:00
Konstantin Lebedev
3a3699867b Status PartialContent for Content-Range response 2021-03-15 23:30:22 +05:00
Konstantin Lebedev
90510e3137 If WriteHeader is not called explicitly, the first call to Write 
will trigger an implicit WriteHeader(http.StatusOK).
WriteHeader: Only one header may be written. Go does not currently!
 2021-03-15 21:22:59 +05:00
Konstantin Lebedev
46b9f5cff4 add debug logging 2021-03-15 19:33:33 +05:00