mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-12-12 16:49:03 +08:00
Code Issues Packages Projects Releases Wiki Activity
6,042 Commits 32 Branches 296 Tags 150 MiB
5df38d610d
Commit Graph

3941 Commits

Author SHA1 Message Date
Chris Lu
3f3268cd1b go fmt 2021-04-14 00:30:16 -07:00
Chris Lu
ca0f07a188 move file reader, entryViewCache to file handle 
reduce file object size
 2021-04-14 00:29:58 -07:00
Chris Lu
5985a7d38d add log level possible values 
fix https://github.com/chrislusf/seaweedfs/issues/1989
 2021-04-12 21:15:51 -07:00
Chris Lu
f5de42fae3
Merge pull request #1975 from kmlebedev/iam_handlers 
IAM handlers
 2021-04-12 12:07:45 -07:00
Chris Lu
1e033d45b8 simpler logic 
related to https://github.com/chrislusf/seaweedfs/pull/1981
 2021-04-12 12:04:53 -07:00
Chris Lu
52200a903b
Merge pull request #1981 from ueni-ltd/if-modified-since 
Fix If-Modified-Since behavior
 2021-04-12 12:02:54 -07:00
Chris Lu
519b0e1e49 filer: upload to a directory without "/" suffix 
fix https://github.com/chrislusf/seaweedfs/issues/1988
 2021-04-12 11:56:56 -07:00
Konstantin Lebedev
8e02e138ea Merge branch 'upstreamMaster' into iam_handlers 2021-04-12 11:22:51 +05:00
Chris Lu
742ab1ec81 2.39 2021-04-11 19:47:11 -07:00
Chris Lu
0df5b53ad8 adjust help message 2021-04-11 00:26:28 -07:00
Chris Lu
f62c153274 go fmt 2021-04-10 23:48:18 -07:00
Chris Lu
af313dff58 add gateway for easier POST and DELETE blobs 2021-04-10 23:47:47 -07:00
Konstantin Lebedev
5021bea698 GetUserPolicy 2021-04-10 23:57:45 +05:00
Chris Lu
0b82edc0d2 filer: avoid stuck uploader 
fix https://github.com/chrislusf/seaweedfs/issues/1980

reverting the file upload batch executor
 2021-04-09 13:05:15 -07:00
Chris Lu
93f4146ffa properly release the view cache 2021-04-09 12:36:39 -07:00
Merlin Gaillard
f952f979d1 filer: return 304 when If-Modified-Since == Last-Modified 2021-04-09 15:04:17 +02:00
Merlin Gaillard
4d4acc715e s3api: handle 304 response code from filer 2021-04-09 12:13:19 +02:00
Chris Lu
6deb647a8f mount: fix possible memory leak 
if many files are read repeatedly, their metadata are accumulated in memory. This fix cleared the metadata after the file is read.
 2021-04-08 19:47:31 -07:00
Konstantin Lebedev
ba175f81b5 add auth aws signV4 2021-04-08 17:40:47 +05:00
Konstantin Lebedev
995ae91007 add DeleteUserPolicy 2021-04-08 11:16:36 +05:00
Philippe Pepiot
42a761ee20
Fix typo in weed filer long help 2021-04-07 22:47:23 +02:00
Chris Lu
3be061994f skip connection reset error 
fix https://github.com/chrislusf/seaweedfs/issues/1971

this is because the connections are pooled but the volume server has reset the connection
 2021-04-07 00:54:13 -07:00
Chris Lu
c5b08bac1b remove mac specific mount options 2021-04-07 00:54:13 -07:00
Konstantin Lebedev
f5f8eec8e2 fix get filerGrpcAddress 2021-04-06 13:53:56 +05:00
Konstantin Lebedev
011e6e90ee Merge branch 'upstreamMaster' into iamapipr 2021-04-06 13:50:33 +05:00
Konstantin Lebedev
ed79baa30f add tests 2021-04-06 13:43:08 +05:00
Chris Lu
2327c0756b fix to avoid loop 2021-04-05 23:24:26 -07:00
Chris Lu
a37eca78cd 2.38 2021-04-05 19:41:54 -07:00
Chris Lu
c131764c34 ensure error is sent back in channel 
fix https://github.com/chrislusf/seaweedfs/issues/1966

avoid shared readErr variable
 2021-04-05 19:40:12 -07:00
Chris Lu
0f64f5b9c8 mount: add readOnly option 
fix https://github.com/chrislusf/seaweedfs/issues/1961
 2021-04-04 21:40:58 -07:00
Chris Lu
6eee200c13 2.37 2021-04-04 18:45:48 -07:00
Chris Lu
fbb82a5c9c skip limiting if limit is zero 2021-04-04 18:38:33 -07:00
Chris Lu
8251d1140e refactor 2021-04-04 18:38:33 -07:00
Chris Lu
5dca98fe71 fix typo 2021-04-03 11:37:13 -07:00
Chris Lu
5e64f65632 ensure tables are created 
fix https://github.com/chrislusf/seaweedfs/issues/1957
 2021-04-02 12:02:26 -07:00
Chris Lu
319743d330 go fmt 2021-04-02 02:22:59 -07:00
Chris Lu
8e404a1433 go fmt 2021-04-02 02:22:26 -07:00
Chris Lu
bdf2ddddfd revert to same implementation as before 
This reverts commit 7e8edc3c4a.
 2021-04-02 02:21:38 -07:00
Chris Lu
67e019d54b add missing changes 2021-04-02 01:11:44 -07:00
Chris Lu
7e8edc3c4a refactoring 2021-04-02 01:10:24 -07:00
Chris Lu
cc0df36a9e smaller file chunks are faster 2021-04-01 02:21:59 -07:00
Chris Lu
cefe66f159 dedicated upload processor 
avoid thundering effect of overloading volume servers
 2021-04-01 02:21:40 -07:00
Chris Lu
1f984d2645 refactor buffer pool 2021-04-01 02:20:00 -07:00
Chris Lu
b5880334fc refactor 2021-03-30 21:07:34 -07:00
Chris Lu
e79e2ddeed remove unused MoveEvents 2021-03-30 20:42:44 -07:00
Chris Lu
e06807a55b adjust logs 2021-03-30 20:36:06 -07:00
Chris Lu
ac875976c0 filer, volume: add concurrent upload size limit to avoid OOM 
add some back pressure when writes are slow
 2021-03-30 02:10:53 -07:00
Chris Lu
a1e18a1384 server add memory profiling 2021-03-30 02:10:53 -07:00
LazyDBA247-Anyvision
9385e2224d fix postgres2 upsertQuery syntax also 2021-03-30 10:51:52 +03:00
Chris Lu
a5662c9e5e adjust ordering 2021-03-30 00:26:57 -07:00
Chris Lu
bec74c3e6a
Merge pull request #1948 from LazyDBA247-Anyvision/master 
add enableUpsert=true
 2021-03-30 00:25:01 -07:00
LazyDBA247-Anyvision
9f1cab179c use a compatible pg syntax for upsert 2021-03-30 10:21:27 +03:00
Chris Lu
a95929e53c reduce default concurrentWriters to 32 2021-03-30 00:17:52 -07:00
LazyDBA247-Anyvision
7f44d953b5 fix GetBool 2021-03-30 01:36:02 +03:00
LazyDBA247-Anyvision
4c51e6a660 add enableUpsert=true 
and rename config to upsertQuery
 2021-03-30 00:32:03 +03:00
Chris Lu
b61e9f6a1a adjust ordering 2021-03-29 00:49:50 -07:00
Chris Lu
1ebdbad222
Merge pull request #1946 from LazyDBA247-Anyvision/master 
Adding Customisation - insertQuery for postgres/mysql
 2021-03-29 00:42:48 -07:00
Konstantin Lebedev
8a95f9c10c iam GetUser 2021-03-29 12:01:44 +05:00
LazyDBA247-Anyvision
96c62bd34d scaffold - insertQuery 
Example for upsert query for mysql & postgres variants
 2021-03-29 10:01:12 +03:00
LazyDBA247-Anyvision
4a02389eb0 Adding custom insertQuery support for postgres/2 mysql/2 2021-03-29 09:58:13 +03:00
Chris Lu
6b7aa9633f 2.36 2021-03-28 19:09:06 -07:00
Chris Lu
bd7471d877 refactor 2021-03-25 12:05:59 -07:00
李海
06be5dc6c3 log snowflake sequencer nodeid's hex when start, it'll be in part of new assigned fid 2021-03-25 18:59:40 +08:00
李海
69b2dab9c6 add a snowflake sequencer as more robust fid generator, but less compressable than small auto-inc id 2021-03-25 18:49:26 +08:00
Konstantin Lebedev
40938d6a47 SaveInsideFiler S3 Configuration 2021-03-25 12:34:11 +05:00
Konstantin Lebedev
9f26f2815c SaveAs S3 Configuration 2021-03-25 12:34:11 +05:00
Konstantin Lebedev
82b0463fac handler PutUserPolicy 
https://docs.aws.amazon.com/IAM/latest/APIReference/API_PutUserPolicy.html
 2021-03-25 12:34:11 +05:00
Konstantin Lebedev
d7719d0542 base handlers 2021-03-25 12:34:11 +05:00
Konstantin Lebedev
03c7953254 init Iam Api Server 2021-03-25 12:34:11 +05:00
Chris Lu
82dfe06066
Merge pull request #1933 from kmlebedev/missingKeysThreshold 
add missingKeysThreshold for fsck
 2021-03-24 10:56:25 -07:00
Konstantin Lebedev
020a5d40c3 avoid counter is null 2021-03-24 22:51:22 +05:00
Konstantin Lebedev
df6cf0a2fa nonRepairThreshold 2021-03-24 22:07:13 +05:00
Konstantin Lebedev
9ce82a9cd2 GOMAXPROCS use defaults to the value of runtime.NumCPU 
https://github.com/chrislusf/seaweedfs/issues/1942
 2021-03-24 18:45:58 +05:00
Konstantin Lebedev
60972f1c97 rename option repairThreshold 2021-03-24 13:24:49 +05:00
Chris Lu
a801332b0d filer: return 409 if file conflicts with a directory 
fix https://github.com/chrislusf/seaweedfs/issues/1938
 2021-03-24 01:03:11 -07:00
Chris Lu
c42b95c596 more help message 
fix https://github.com/chrislusf/seaweedfs/issues/1937
 2021-03-23 17:27:57 -07:00
Konstantin Lebedev
40a9e88c07 rename var 2021-03-23 21:42:35 +05:00
Konstantin Lebedev
d21a5bf5d0 add missingKeysThreshold for fsck 2021-03-23 15:04:07 +05:00
Chris Lu
85cff10787 mysql: ensure name column is case sensitive 
fix https://github.com/chrislusf/seaweedfs/issues/1931
 2021-03-23 00:46:50 -07:00
Chris Lu
4abb511db3 make a local copy of the in memory cached data 2021-03-22 22:33:07 -07:00
Chris Lu
3cbc40fa48 avoid creating multiple reader 2021-03-22 22:32:47 -07:00
Chris Lu
2f7c7afdec refactor: remove unused parameter 2021-03-22 22:13:19 -07:00
Chris Lu
d48dd0c738 use []byte directly instead of bytes.Buffer 2021-03-22 22:12:57 -07:00
Chris Lu
8eb14967ef shell: volume.check.disk adds optional slow mode 
fix https://github.com/chrislusf/seaweedfs/issues/1930
 2021-03-22 21:01:43 -07:00
Chris Lu
288369cfc7 mount: release resources when Forget() is called 
address https://github.com/chrislusf/seaweedfs/issues/1929
 2021-03-22 16:22:19 -07:00
Chris Lu
5d931eff27 avoid possible nil 
fix https://github.com/chrislusf/seaweedfs/issues/1928

The nil was because of `dn.Parent().UnlinkChildNode(dn.Id())` in topo.UnRegisterDataNode() function, when the dn leaves the cluster.
 2021-03-22 13:24:07 -07:00
Chris Lu
f315eb2bb8 fix build error 2021-03-22 00:12:53 -07:00
Chris Lu
1dd5bc134c 2.35 2021-03-22 00:05:09 -07:00
Chris Lu
b465095db1 shell: add volume.check.disk to fix inconsistency for replicated volumes 
fix https://github.com/chrislusf/seaweedfs/issues/1923
 2021-03-22 00:03:16 -07:00
Chris Lu
df461402cc ensure entry attribute is not nill 
fix https://github.com/chrislusf/seaweedfs/issues/1926
 2021-03-21 22:27:30 -07:00
Chris Lu
102a951377 refactor, split into 2 files 2021-03-21 13:05:53 -07:00
Chris Lu
f6e2566ee5 better logs for cookie mismatch 2021-03-20 12:02:57 -07:00
Chris Lu
27e24162f2 minor 2021-03-20 06:37:03 -07:00
Chris Lu
41cc6e661b fs.mv ensures there are 2 arguments 2021-03-20 06:34:13 -07:00
Chris Lu
b2c1c209a5 refactor 2021-03-20 06:34:13 -07:00
Chris Lu
e9b1853d63 go fmt 2021-03-19 01:31:56 -07:00
Chris Lu
b1a86cf808 s3: copy object to itself 
fix https://github.com/chrislusf/seaweedfs/issues/1922
 2021-03-19 01:31:49 -07:00
Konstantin Lebedev
bf94149920 add line 2021-03-17 23:41:34 +05:00
Konstantin Lebedev
c5705e7a35 dir + slash key does not exist 
https://github.com/chrislusf/seaweedfs/issues/1917
 2021-03-17 23:40:42 +05:00
Konstantin Lebedev
eb54993a4e Merge branch 'upstreamMaster' into check_chunkviews_mr 
# Conflicts:
#	weed/filer/filechunk_manifest.go
#	weed/filer/stream.go
#	weed/replication/repl_util/replication_util.go
#	weed/util/fasthttp_util.go
 2021-03-16 15:29:49 +05:00
Chris Lu
9672f9e1b2 2.34 2021-03-16 03:01:15 -07:00
Chris Lu
69694a17be reverting 7d57664c2d 2021-03-16 02:59:26 -07:00
Chris Lu
c00dd5e62e report error if entry not found 2021-03-16 02:55:49 -07:00
Konstantin Lebedev
06da02739d CheckAllChunkViews() for HEAD requests only 2021-03-16 14:15:17 +05:00
Chris Lu
91a3ac9731 2.33 2021-03-16 00:36:06 -07:00
Chris Lu
4b1ed227d1 revert fasthttp changes 
related to https://github.com/chrislusf/seaweedfs/issues/1907
 2021-03-16 00:33:14 -07:00
Chris Lu
10164d0386
Merge pull request #1905 from wuh-fnst/s3ListCounterErr 
s3: count correction for the number of files in the dir
 2021-03-15 20:54:58 -07:00
Chris Lu
22a07a12d2 [mount] [regression] filer.path option results in empty mount in 2.32 
fix https://github.com/chrislusf/seaweedfs/issues/1906
 2021-03-15 20:46:25 -07:00
wuh-fnst
050a7f9599 s3: count correction for the number of files in the dir 2021-03-16 09:22:03 +08:00
Konstantin Lebedev
3a3699867b Status PartialContent for Content-Range response 2021-03-15 23:30:22 +05:00
Konstantin Lebedev
90510e3137 If WriteHeader is not called explicitly, the first call to Write 
will trigger an implicit WriteHeader(http.StatusOK).
WriteHeader: Only one header may be written. Go does not currently!
 2021-03-15 21:22:59 +05:00
Konstantin Lebedev
46b9f5cff4 add debug logging 2021-03-15 19:33:33 +05:00
Konstantin Lebedev
7194a5e7bf avoid http error: superfluous response.WriteHeader 
https://github.com/chrislusf/seaweedfs/issues/1838
 2021-03-15 18:52:59 +05:00
Chris Lu
9f00f95bfb 2.32 2021-03-14 21:32:00 -07:00
Chris Lu
a4cfffc264 shell: fix moving volume, volume server evacuate 
fix https://github.com/chrislusf/seaweedfs/issues/1534
 2021-03-14 21:29:55 -07:00
Chris Lu
ca100568f7 fix parsing 2021-03-14 20:50:14 -07:00
Chris Lu
352ba23f83 revert previous change 
revert 29e62aba00
 2021-03-14 20:49:56 -07:00
Chris Lu
29e62aba00 possible fix for volume balance 
address https://github.com/chrislusf/seaweedfs/issues/1534
 2021-03-14 20:23:19 -07:00
Chris Lu
35d939a1d2 go fmt 2021-03-14 13:21:02 -07:00
Chris Lu
2d4c2db81d filer: leveldb, rocksdb auto create store directory 
fix https://github.com/chrislusf/seaweedfs/issues/1901
 2021-03-14 13:20:14 -07:00
Chris Lu
6d3a96eb56 filer: mysql2, postgres2 trigger actions on bucket creation and deletion 
fix https://github.com/chrislusf/seaweedfs/issues/1877
 2021-03-13 22:07:39 -08:00
Chris Lu
cb423312a4 prevent nil volume message 2021-03-13 11:05:29 -08:00
Chris Lu
972327f966 prevent nil volume nm 2021-03-13 11:04:51 -08:00
Chris Lu
2a68ddb963 default to empty host to bind to tcp4 and tcp6 2021-03-12 15:32:51 -08:00
Chris Lu
5fbcaaf37a fullpath() test is not needed 2021-03-12 09:45:25 -08:00
Chris Lu
7d57664c2d mount: internals switch to filer.Entry instead of protobuf 2021-03-12 00:36:38 -08:00
wuh-fnst
4c1d945e46 make List correctly judge whether it is the last file 2021-03-12 11:22:26 +08:00
Chris Lu
cca66c7fbe print out the url 2021-03-11 18:39:45 -08:00
Chris Lu
7f887c78d2 fix 2021-03-11 18:18:16 -08:00
Chris Lu
fcc52d1e4e shell: add s3.clean.uploads to clean up stale multipart uploads 
fix https://github.com/chrislusf/seaweedfs/issues/1855
 2021-03-11 18:05:24 -08:00
Chris Lu
40dc5ac904 mount: add a bit retry when connecting to filer during startup 2021-03-11 16:36:43 -08:00
Chris Lu
aba47fd9e8 reduce repeated calls to dir.FullPath() 2021-03-11 14:08:20 -08:00
Chris Lu
d084334ffd
Merge pull request #1883 from kmlebedev/passDelNonEmptyFolder 
s3 delete-objects pass OK for fail to delete non-empty folder
 2021-03-11 11:01:44 -08:00
Chris Lu
0e35836cc1
Merge pull request #1890 from kmlebedev/ZeroDataSize 
error if read chunk zero data size
 2021-03-11 10:51:48 -08:00
Konstantin Lebedev
58cdcc6d6e error if urls not found 2021-03-11 23:34:36 +05:00
Chris Lu
79280e1513 escape special characters 
fix https://github.com/chrislusf/seaweedfs/issues/1884
 2021-03-11 09:49:40 -08:00
Konstantin Lebedev
4db21012c1 error if read chunk zero data size 2021-03-11 22:38:59 +05:00
wuheng
828fbf3fb7 s3: "isLast" returns true when the file does not exist 2021-03-11 15:20:50 +08:00
Chris Lu
5d446673a6 fix error printing 2021-03-10 17:13:04 -08:00
Chris Lu
ca546f47e1 s3: escape object key if containing special characters 
fix https://github.com/chrislusf/seaweedfs/issues/1884
 2021-03-10 13:19:28 -08:00
Chris Lu
105fcd7bfc
Merge pull request #1872 from kmlebedev/grpc_auth 
TLS allowed CommonNames
 2021-03-10 11:17:49 -08:00
Konstantin Lebedev
bb3af2d70b no directoriesWithDeletion 2021-03-10 19:52:41 +05:00
Konstantin Lebedev
15b018da34 s3 delete-objects pass OK for fail to delete non-empty folder 2021-03-10 19:41:35 +05:00
Konstantin Lebedev
348e21a08c add comments 2021-03-10 14:42:39 +05:00
Konstantin Lebedev
831953c55c allowed wildcard domain 2021-03-10 14:02:13 +05:00
Konstantin Lebedev
4bf93d6e63 comma-separated 2021-03-10 12:43:13 +05:00
Konstantin Lebedev
0e02f7e258 comma-separated SSL certificate common names 2021-03-10 12:42:44 +05:00
Chris Lu
737bde5ab7 refactoring, remove duplicated code 2021-03-09 23:23:01 -08:00
Chris Lu
5ba4b479f8 properly lock file.entry object 
fix https://github.com/chrislusf/seaweedfs/issues/1882
 2021-03-09 23:08:38 -08:00
Chris Lu
5be83dd4a9 add comments 2021-03-09 14:13:48 -08:00
Chris Lu
9d8ca513d9 refactor 2021-03-09 13:21:26 -08:00
Chris Lu
f5b5e4da2d 2.31 2021-03-09 12:52:16 -08:00
Chris Lu
828f6e9f4d volume: auto add missing vif files 
fix https://github.com/chrislusf/seaweedfs/issues/1878
 2021-03-09 12:09:32 -08:00
Chris Lu
387c6f4218 Revert "delete the folder object in multi_object_delete" 
This reverts commit 2e89c8c9ae.
 2021-03-09 10:07:27 -08:00
qieqieplus
5b16820924 fix: collection stats won't update if all volumes expired at same iteration 2021-03-09 18:49:45 +08:00
Konstantin Lebedev
190fada1ef TLS allowed commonNames 2021-03-08 21:39:44 +05:00
Konstantin Lebedev
c6d3735605 permitCommonNames 
https://github.com/chrislusf/seaweedfs/issues/1841
https://jbrandhorst.com/post/grpc-auth/
 2021-03-08 13:16:17 +05:00
Chris Lu
d888e9a90f 2.30 2021-03-07 15:01:19 -08:00
Chris Lu
60686a0bb7 make tcp optional 2021-03-07 14:45:36 -08:00
Chris Lu
3b2a9c98ef fix nil 2021-03-07 11:45:35 -08:00
Chris Lu
df72dc206d stats collect chunk upload retry count 2021-03-07 11:26:15 -08:00
Chris Lu
726edab054 avoid nil when closing an index 
fix https://github.com/chrislusf/seaweedfs/issues/1870
 2021-03-07 11:03:09 -08:00
Chris Lu
bdfed16d42 avoid nil exception 
fix https://github.com/chrislusf/seaweedfs/issues/1869
 2021-03-07 01:49:06 -08:00
Chris Lu
1bd880dcdb adds tcp writes benchmark 2021-03-06 14:26:27 -08:00
Chris Lu
9d402ebe9f refactoring 2021-03-06 14:26:27 -08:00
Chris Lu
e9eddfb8a2 remove wrong writes 2021-03-06 14:26:27 -08:00
Chris Lu
8363be8548 trim out trailing return character 2021-03-06 14:26:27 -08:00
Chris Lu
38fc200e56 CRCWriter consistent with CRC 2021-03-06 14:26:27 -08:00
Chris Lu
1444e9d275 migrated multi host connection pool from godropbox package 
removing unneeded dependencies, which involved etcd versions.
 2021-03-06 14:26:27 -08:00
Patrick Schmidt
7413d59750 Fix EC shard count logic 
This fixes the calculation of the amount of EC shards a node holds.
Previously a global counter was increased, but also used inside the
loop to apply disk usage deltas. This led to wrong absolute numbers.
The fix is to apply only deltas of single EC shards per iteration.
 2021-03-05 12:50:58 +01:00
Chris Lu
400de380f4 volume server: support tcp direct put/get/delete 2021-03-05 02:29:38 -08:00
Chris Lu
2e89c8c9ae delete the folder object in multi_object_delete 
fix https://github.com/chrislusf/seaweedfs/issues/1857
 2021-03-04 14:03:40 -08:00
Chris Lu
c5cb346a08 fix compilation error 2021-03-04 12:36:20 -08:00
Chris Lu
f5cc96f289 skip checking master peers if not starting master 2021-03-04 12:31:26 -08:00
Chris Lu
40ff30b83f adjust help message 2021-03-03 02:12:46 -08:00
Chris Lu
c0842fe99f add filer.meta.backup command 2021-03-03 02:02:29 -08:00
Chris Lu
0c5c51eb98 adjust help message 2021-03-02 23:07:39 -08:00
Chris Lu
bd727b7b53 go fmt 2021-03-02 20:59:56 -08:00
Chris Lu
e982b06bcd support IPv6 2021-03-02 20:59:39 -08:00
Chris Lu
bcf32591b7 reduce possibility of nil entry 2021-03-02 13:33:56 -08:00
Chris Lu
be9c7c21ec reduce possibility of nil file entry 2021-03-02 11:17:17 -08:00
Chris Lu
02e146f3e1 return err 2021-03-02 08:54:18 -08:00
Chris Lu
5511722420 s3: list permission is needed to list my buckets 
fix https://github.com/chrislusf/seaweedfs/issues/1837
 2021-03-01 12:41:55 -08:00
Chris Lu
c46d16b2f3 skip network error when remote volumes start 2021-03-01 01:20:06 -08:00
Chris Lu
6e43e8ce17 add logs for remote file loading 2021-03-01 00:48:59 -08:00
Chris Lu
015d16f43f add vif file versions in case loading superblock fails 2021-03-01 00:48:30 -08:00
Chris Lu
d680676d45 skip already loaded backends 2021-03-01 00:47:03 -08:00
Chris Lu
540441fd38 go fmt 2021-02-28 20:34:14 -08:00
Chris Lu
c9722dceb2 adjust log level 2021-02-28 20:26:55 -08:00
Chris Lu
1b1c018165 adjust text 2021-02-28 19:02:43 -08:00
Chris Lu
2cca07b44c cloud tier to non-AWS s3 gateways 2021-02-28 18:59:09 -08:00
Chris Lu
f29ae0db76 2.29 2021-02-28 18:08:26 -08:00
bingoohuang
5cdff56731 set default env prefix to WEED_ 2021-03-01 09:35:23 +08:00
Chris Lu
e52c94640e filer.backup: added to replace filer.replicate 2021-02-28 16:22:27 -08:00
Chris Lu
014a31d11a minor 2021-02-28 16:21:09 -08:00
Chris Lu
f2fcb77808 local file sink: create backup files with permission 0755 2021-02-28 16:20:47 -08:00
Chris Lu
984fdd6192 always use non bucket prefixing url 2021-02-28 16:20:13 -08:00
Chris Lu
9abb041763 filer source: support filerProxy mode 2021-02-28 16:19:47 -08:00
Chris Lu
678c54d705 data sink: add incremental mode 2021-02-28 16:19:03 -08:00
Chris Lu
4ff2c5c4c9 rename file 2021-02-28 16:14:21 -08:00
Chris Lu
3b76a51f5f add help message 2021-02-27 12:15:49 -08:00
Chris Lu
70434df105 use "options" 2021-02-27 12:12:53 -08:00
Chris Lu
d3c31c69a7 avoid confusion: conf and config are too similar 2021-02-27 12:08:09 -08:00
Chris Lu
543c8daa6d
Merge pull request #1844 from kmlebedev/processRangeRequest 2021-02-26 09:18:22 -08:00
Patrick Schmidt
5f7b024891 Show the real disk usage in stats calls 
Currently the file size of only one volume location is taken into
account in the stats. This commit multiplies the disk usages by the
amount of nodes holding a replica of the volume.
This will yield the expected amount of disk usage and matches the
total size calculations from before.
 2021-02-26 13:58:40 +01:00
Konstantin Lebedev
ee21c0042e log error write entry 2021-02-26 14:18:01 +05:00
Chris Lu
7ba75e3d5a filer: do not return no content for empty files 
fix https://github.com/chrislusf/seaweedfs/issues/1831
fix https://github.com/chrislusf/seaweedfs/issues/1830
 2021-02-23 12:25:27 -08:00
Chris Lu
9edd964627 volume.tier.move: avoid repeated move for replicated volumes 
fix https://github.com/chrislusf/seaweedfs/issues/1792#issuecomment-784139348
 2021-02-23 03:49:14 -08:00
Chris Lu
37f104f88f 2.28 2021-02-22 22:54:34 -08:00
Chris Lu
90cdf9dcac avoid conflict with "weed scaffold -config=xxx" 2021-02-22 16:57:27 -08:00
Chris Lu
72b0d9d8c4 avoid unnecessary user home checking 2021-02-22 16:50:56 -08:00
Chris Lu
2270737344 volume: avoid fixed vacuum timeout for large volumes 
1GB for 3 minutes, about 5.7MB/s
 2021-02-22 12:52:37 -08:00
Chris Lu
44bdfb2d15 filer: avoid encryption and compression at the same time 
fix https://github.com/chrislusf/seaweedfs/issues/1828
 2021-02-22 12:22:49 -08:00
Chris Lu
62191b08ea disk type support custom tags 2021-02-22 02:03:12 -08:00
Chris Lu
30b30b8fe0 volume.tier.move: passing non-empty disk type 2021-02-22 01:59:03 -08:00
Chris Lu
5da63e045e avoid moving to another server with the same volume id 2021-02-22 01:44:18 -08:00
Chris Lu
6a4546d2c0 shell: add volume.tier.move 2021-02-22 01:30:07 -08:00
Chris Lu
1c233ad986 refactoring 2021-02-22 00:28:42 -08:00
Chris Lu
151c281f36 2.27 2021-02-21 19:29:27 -08:00
Chris Lu
258e93bc86
Merge pull request #1823 from bingoohuang/master 
feature: support command line arguments in a configuration file
 2021-02-20 22:35:18 -08:00
Chris Lu
03c643aa6b fix test 
fileSize need to be divided by 8
 2021-02-20 21:19:21 -08:00
Chris Lu
7635f6b9fa disk file avoid file.Stat() 2021-02-20 20:06:06 -08:00
bingoo
1af6c96cd0 fix log help 2021-02-21 10:46:01 +08:00
Chris Lu
a2383b3b12 Revert "Revert "avoid file.Stat()"" 
This reverts commit 7ef4c24f18.
 2021-02-20 12:42:09 -08:00
Chris Lu
7ef4c24f18 Revert "avoid file.Stat()" 
This reverts commit 98c93ca465.
 2021-02-20 12:39:33 -08:00
Chris Lu
1ad3200094 skip seek() when index file writes 2021-02-20 12:39:25 -08:00
Chris Lu
98c93ca465 avoid file.Stat() 
avoid one Syscall, but did not help on performance though
 2021-02-20 09:44:17 -08:00
bingoohuang
ee082ae402 revert comment temporarily 2021-02-20 16:59:38 +08:00
bingoohuang
ee7cdf3668 revert ErrXyz to ErrorXyz temporarily. 2021-02-20 16:58:08 +08:00
bingoohuang
50df484d86 support command line arguments in a configuration file so we can weed master -config=master.conf, weed volume -config=volume.conf and etc. 2021-02-20 16:52:57 +08:00
bingoohuang
f69356f589
Merge pull request #3 from chrislusf/master 2021-02-20 16:45:02 +08:00
Chris Lu
565f7a6e72 Update data_node.go 2021-02-19 14:22:36 -08:00
Chris Lu
a37473ae60 add back volume ids 
address https://github.com/chrislusf/seaweedfs/issues/1792#issuecomment-782339576
 2021-02-19 14:22:12 -08:00
Chris Lu
a1210d1e8d Revert "volume: avoid file.stat(), file.seek() if possible during writes" 
This reverts commit c78409a598.
 2021-02-19 03:56:27 -08:00
Chris Lu
b961cd6208 add WIP message 2021-02-19 03:39:19 -08:00
Chris Lu
f465d63b5d grpc should fail when heart beating to master 
fix https://github.com/chrislusf/seaweedfs/issues/1820
 2021-02-19 03:37:29 -08:00
Chris Lu
c576ad04ac fix volume server display for volumes 2021-02-19 01:38:56 -08:00
bingoohuang
7ffe736d20 renaming to ErrXyz 2021-02-19 15:59:55 +08:00
Chris Lu
c78409a598 volume: avoid file.stat(), file.seek() if possible during writes 2021-02-18 23:22:54 -08:00
Chris Lu
29575dadc5 remove extra files 2021-02-18 19:14:58 -08:00
Chris Lu
73958e357d add descriptive error if no free volumes 2021-02-18 19:10:20 -08:00
Chris Lu
776f497469 filer: fs.configure should try to read from entry.content also 
related to https://github.com/chrislusf/seaweedfs/issues/1792
 2021-02-18 17:07:02 -08:00
Chris Lu
e1992c83de webdav adds disktype 2021-02-18 15:45:44 -08:00
Chris Lu
c2ad6f1047 webdav add replication setting 
fix https://github.com/chrislusf/seaweedfs/issues/1817
 2021-02-18 12:15:09 -08:00
Chris Lu
816ef0ed30 add more help message 
fix https://github.com/chrislusf/seaweedfs/issues/1816
 2021-02-18 10:34:49 -08:00
bingoohuang
352ac2f271 Merge remote-tracking branch 'origin/master' 2021-02-18 14:05:51 +08:00
bingoohuang
eab6e31d34 use backticks instead of double quotes to avoid escaped additionally in regex 2021-02-18 14:05:28 +08:00
Chris Lu
3575d41009 go fmt 2021-02-17 20:57:08 -08:00
Chris Lu
6daa932f5c refactoring to get master function, instead of passing master values directly 
this will enable retrying later
 2021-02-17 20:55:55 -08:00
Chris Lu
dd9f3a0104 add sleep between upload retries 2021-02-17 20:55:11 -08:00
Chris Lu
aa17311063 s3: set filename when uploading file chunks 2021-02-17 20:54:53 -08:00
Chris Lu
3f8b0da677 filer: do not print password on error 
fix https://github.com/chrislusf/seaweedfs/issues/1809
 2021-02-17 02:13:52 -08:00
Chris Lu
cd866664a8 skip JWT if fileId is empty 
related to https://github.com/chrislusf/seaweedfs/issues/1808
 2021-02-16 15:39:12 -08:00
Chris Lu
632c94d438 Merge branch 'extend_to_disk_type' 2021-02-16 11:06:13 -08:00
Chris Lu
0611233f16 adjust printing 2021-02-16 10:55:30 -08:00
Chris Lu
68775d29e3 fix tests 2021-02-16 10:51:03 -08:00
Chris Lu
7403cd43c5 only use "" for hdd to avoid two values for the same thing 2021-02-16 10:50:42 -08:00
Chris Lu
b314d78e97 fix print 2021-02-16 10:48:28 -08:00
Chris Lu
53ca7e66ef avoid dead lock 2021-02-16 10:48:16 -08:00
Chris Lu
38efc6f572 simplify 2021-02-16 10:47:32 -08:00
Chris Lu
ad70107eb4 ec balance among nodes with HDD 2021-02-16 06:22:50 -08:00
Chris Lu
3097b9a9b7 fix existence checking 2021-02-16 05:59:43 -08:00
Chris Lu
ebf320ec22 avoid divided by zero 2021-02-16 05:59:24 -08:00
Chris Lu
a0c6db361c avoid nil 2021-02-16 05:33:38 -08:00
Chris Lu
36f95e50a9 avoid possible nil disk info 2021-02-16 05:13:48 -08:00
Chris Lu
43101ccea0 move to the empty nodes first 2021-02-16 04:27:16 -08:00
Chris Lu
38bbef7ec1 avoid nil 2021-02-16 04:16:46 -08:00
Chris Lu
cb9cc29518 volume.list display; fix updating maxVolumeCount for disk 2021-02-16 03:55:24 -08:00
Chris Lu
3fe628f04e use hdd instead of empty string 2021-02-16 03:03:00 -08:00
Chris Lu
f8446b42ab this can compile now!!! 2021-02-16 02:47:02 -08:00
Chris Lu
b9b5b932c5 filer: postgres2 mysql2 avoid repeatedly creating tables 2021-02-15 16:19:24 -08:00
Chris Lu
71f0c19515 2.26 2021-02-15 13:38:38 -08:00
Chris Lu
8b169aa47c s3: Critical - S3 Delete Multiple Objects, delete the bucket 
fix https://github.com/chrislusf/seaweedfs/issues/1806
 2021-02-15 13:38:29 -08:00
Chris Lu
50202d1391
Merge pull request #1805 from LazyDBA247-Anyvision/master 
better postgresql connection pool management
 2021-02-14 22:26:29 -08:00
Chris Lu
8726ef34c1 2.25 2021-02-14 22:23:43 -08:00
LazyDBA247-Anyvision
7f458d5e78 better postgres connection pool management 
adding SetConnMaxLifetime configuration (https://golang.org/pkg/database/sql/#DB.SetConnMaxLifetime)
to enable refresh of stale connections.
 2021-02-15 07:45:09 +02:00
Chris Lu
0bc3a1f9e8 disk type only supports hdd and ssd, not ready for random tags yet 2021-02-14 11:38:43 -08:00
LazyDBA247-Anyvision
51b4963e2e postgres2 & memsql2 
add escape (quote identifiers) for the dynamic sql
so tables (collections) with special characters will work.
 2021-02-14 13:14:36 +02:00
Chris Lu
876dbe3d26 collect disk types 2021-02-13 23:25:16 -08:00
Chris Lu
861f8fab9b refactoring 2021-02-13 22:34:12 -08:00
Chris Lu
881f05f5a0 adjust help message 2021-02-13 16:13:52 -08:00
Chris Lu
ef76365ec2 adjust help message 2021-02-13 15:47:08 -08:00
Chris Lu
7ce647f27e support customizable disk type 2021-02-13 15:42:42 -08:00
Chris Lu
4bd8a692d8 disk type can be generic tags 2021-02-13 13:50:14 -08:00
Chris Lu
712b3e9e53 mount: release need to avoid nil reader 
fix https://github.com/chrislusf/seaweedfs/issues/1803
 2021-02-13 13:39:37 -08:00
Chris Lu
4ce56bac08 fix format error 2021-02-12 06:44:26 -08:00
Chris Lu
0604afcda7 mount: remove unnecessary flush on release 2021-02-12 06:01:04 -08:00
Chris Lu
7d9dc3c6a2 use fasthttp lib to read 2021-02-12 05:59:52 -08:00
Chris Lu
487e435679 adjust http max idle connections per host 
related to https://github.com/chrislusf/seaweedfs/issues/1802
 2021-02-12 03:47:15 -08:00
Chris Lu
0f426ce34d
Merge pull request #1801 from kmlebedev/recoveringRabbitMQ 
Do reconnect to RabbitMQ
 2021-02-11 01:05:03 -08:00
Konstantin Lebedev
94eac4f00e Do reconnect to RabbitMQ 
https://github.com/google/go-cloud/issues/2958
https://github.com/chrislusf/seaweedfs/issues/1773
 2021-02-11 13:59:36 +05:00
Chris Lu
885ca34748 volume: fail fast if idx files are missing 
fix https://github.com/chrislusf/seaweedfs/issues/1796
 2021-02-11 00:44:40 -08:00
Chris Lu
a0e84c4fbc go fmt 2021-02-10 23:41:05 -08:00
Chris Lu
c959aaa19b mount: load directory based on correct filer path 
fix https://github.com/chrislusf/seaweedfs/issues/1799
 2021-02-10 23:14:18 -08:00
Konstantin Lebedev
0cfed8c3cb restart replication if RabbitMQ connection closed 
https://github.com/google/go-cloud/issues/2958
 2021-02-10 15:29:05 +05:00
Chris Lu
770393a48c volume: add capability to change disk type when moving a volume 2021-02-09 23:58:08 -08:00
Chris Lu
821c46edf1 Merge branch 'master' into support_ssd_volume 2021-02-09 11:37:07 -08:00
Chris Lu
a833021132 fix refactoring left over 2021-02-07 23:03:03 -08:00
Chris Lu
5e4b5109dd 2.24 2021-02-07 22:29:43 -08:00
Chris Lu
15c60cbb26 close the grpc connection after 10 hours 
related to https://github.com/chrislusf/seaweedfs/issues/1782
 2021-02-07 03:50:01 -08:00
bingoohuang
7256902fb0 fix typo offset.ToAcutalOffset to offset.ToActualOffset 2021-02-07 12:11:51 +08:00
bingoohuang
94ea3bd3a5 renaming NeedleMapType to NeedleMapKind 2021-02-07 09:00:03 +08:00
Chris Lu
c3af72d950 fier store: fix elastic search regression 
fix https://github.com/chrislusf/seaweedfs/issues/1774
 2021-02-04 01:30:14 -08:00
Konstantin Lebedev
2fb4c60540 fix log warn 2021-02-03 21:34:32 +05:00
Konstantin Lebedev
24ec17219b RabbitMQ delay retry with Dead Letter Exchange 
https://github.com/chrislusf/seaweedfs/issues/1773

https://github.com/google/go-cloud/issues/2952
 2021-02-03 21:32:30 +05:00
Chris Lu
a331bbb3ae filer: should return 204 on DELETE to nonexistent file 
related to

https://github.com/chrislusf/seaweedfs/issues/1776
https://github.com/chrislusf/seaweedfs/issues/1160
 2021-02-03 00:40:31 -08:00
Chris Lu
5353e38469 S3: should return 204 on DELETE to nonexistent file 
fix https://github.com/chrislusf/seaweedfs/issues/1776
 2021-02-03 00:35:44 -08:00
Chris Lu
2396ac234c filer: fix elastic search pagination 
possible fix for https://github.com/chrislusf/seaweedfs/issues/1774
 2021-02-02 11:34:12 -08:00
Chris Lu
609daaf387 s3: DeleteMultipleObjectsHandler clean up leftover empty folders 
fix https://github.com/chrislusf/seaweedfs/issues/1772
 2021-02-01 10:49:17 -08:00
Chris Lu
d67ccb66c0 webdav: can start together with "weed server" or "weed filer" 2021-01-31 22:16:52 -08:00
Chris Lu
76430790b9 webdav: cache to version specific folder 2021-01-31 22:02:03 -08:00
Chris Lu
318a3d2efc 2.23 2021-01-31 20:18:06 -08:00
Chris Lu
4be51c0701 filer: leveldb and hbase may miss files when listing large directories more than 1024 
fix https://github.com/chrislusf/seaweedfs/issues/1768
 2021-01-31 20:11:44 -08:00
Chris Lu
1102ae32c4 fix concurrent map reads 2021-01-31 18:26:26 -08:00
Chris Lu
314dc1c957 filer: etcd store fix listing 
fix https://github.com/chrislusf/seaweedfs/issues/1767
 2021-01-30 13:38:44 -08:00
Chris Lu
d475c89fcc go fmt 2021-01-28 15:23:46 -08:00
Chris Lu
19295600f9 mount: change option name to volumeServerAccess, with publicUrl and filerProxy modes 2021-01-28 15:23:16 -08:00
Chris Lu
990fa69bfe add back AdjustedUrl() related code 2021-01-28 14:36:29 -08:00
Chris Lu
9292796ec2 fmt 2021-01-28 14:29:14 -08:00
Chris Lu
c2bf1a88ac delete from the deepest directory first when checking empty folders 2021-01-28 14:28:40 -08:00
Chris Lu
e9d8201925 s3: batch purge empty folders 2021-01-28 13:20:06 -08:00
Chris Lu
cf252fc0cd mount: report error when Flush() 
The error in Release() is not working.

See https://github.com/jaderhs/libfuse/blob/master/FAQ

related to https://github.com/chrislusf/seaweedfs/issues/1765
 2021-01-28 04:46:37 -08:00
Chris Lu
9a06c35da4 replicate: incremental sink only contains new and updated files 
address da08402ba2
 2021-01-28 02:39:22 -08:00
Chris Lu
da08402ba2 replicate: use creation time for local incremental file sink 
related to https://github.com/chrislusf/seaweedfs/pull/1762
 2021-01-28 02:17:41 -08:00
Chris Lu
822f1ade9d
Merge pull request #1762 from kmlebedev/backupsink 
replication to create time date directory
 2021-01-28 02:03:33 -08:00
Konstantin Lebedev
be1062b7fc rename in scaffold 2021-01-28 14:59:20 +05:00
Konstantin Lebedev
02fdc0a333 rename backup to local_incremental and use mtime 2021-01-28 14:56:13 +05:00
Chris Lu
5f72c388bc
Merge pull request #1764 from danlsgiga/handle-umask 
Use provided umask on mount
 2021-01-28 01:09:20 -08:00
Daniel Santos
d38613c49a match upstream 2021-01-27 20:56:41 -07:00
Chris Lu
b81956bcb5 mount: invalidate kernel cache when mounted to a filer path 
fix https://github.com/chrislusf/seaweedfs/issues/1752#issuecomment-768178422
 2021-01-27 10:28:37 -08:00
Konstantin Lebedev
6b54ff9912 replication to create time date directory 2021-01-27 15:01:33 +05:00
Chris Lu
f20ec82a28 update help message 
see https://github.com/chrislusf/seaweedfs/issues/1758#issuecomment-768123224
 2021-01-27 00:48:31 -08:00
Chris Lu
1045ab8b07 avoid error message 2021-01-27 00:21:59 -08:00
Chris Lu
0488c84bea adjust help message 2021-01-27 00:17:52 -08:00
Konstantin Lebedev
3634811408 replication to todays date directory 2021-01-27 12:45:58 +05:00
Chris Lu
5e07afb0f0
Merge pull request #1759 from kmlebedev/sink.local 
replication to local disk storage
 2021-01-26 22:31:54 -08:00
Chris Lu
b3f66199db shell: volume.fix.replication adds collectionPattern with wildcard characters 
fix https://github.com/chrislusf/seaweedfs/issues/1758
 2021-01-26 22:30:39 -08:00
Daniel Santos
0c64d60fdb Fix log message with correct mode 2021-01-26 21:45:52 -07:00
Daniel Santos
0aaaa9b470 Fix var assignment 2021-01-26 21:06:08 -07:00
Daniel Santos
5a761bb7a4 Enforce umask correctly 2021-01-26 20:42:10 -07:00
Chris Lu
ad2a20c8a5 notification add ack and nack 2021-01-26 11:08:44 -08:00
Konstantin Lebedev
612b7975a1 replication to local disk storage 2021-01-26 22:50:25 +05:00
Chris Lu
3a1d3d3413 mount: properly invalidate kernel node cache entry 
fix https://github.com/chrislusf/seaweedfs/issues/1752
 2021-01-26 02:50:53 -08:00
Chris Lu
cc1f3907ff filer.sync: default filerProxy to true 2021-01-25 10:14:42 -08:00
Chris Lu
62ebb917e0 mount: avoid the need to specify different cache directory for different mount 
fix https://github.com/seaweedfs/seaweedfs-csi-driver/issues/11
 2021-01-25 09:10:20 -08:00
Chris Lu
17f7c1c43f Merge branch 'mount_from_outside_cluster' 2021-01-24 19:03:02 -08:00
Chris Lu
00707ec00f mount: outsideContainerClusterMode proxy through filer 
Running mount outside of the cluster would not need to expose all the volume servers to outside of the cluster. The chunk read and write will go through the filer.
 2021-01-24 19:01:58 -08:00
Chris Lu
9bcb28a3ea avoid possible nil reader 
fix https://github.com/chrislusf/seaweedfs/issues/1754
 2021-01-24 17:19:35 -08:00
Chris Lu
2c5eac5705 2.22 2021-01-24 14:06:09 -08:00
Chris Lu
6ca10725b8 Revert "mount: when outside cluster network, use filer as proxy to access volume servers" 
This reverts commit 096e088d7b.
 2021-01-24 03:15:19 -08:00
Chris Lu
096e088d7b mount: when outside cluster network, use filer as proxy to access volume servers 2021-01-24 01:41:38 -08:00
Chris Lu
ccbdb38c89 s3: fix for listing objects if more than 1000 in the folder 
many existing tools have max key set to 1000
 2021-01-24 00:27:52 -08:00
Chris Lu
80b8692688 filer.sync: replicate outside of either cluster, only need to see filers 2021-01-24 00:01:44 -08:00
Chris Lu
f8dbb03bdd filer: append operation returns final file size 2021-01-23 04:19:49 -08:00
Chris Lu
759482e2ac filer: proxy all http methods to volume servers 2021-01-23 03:52:38 -08:00
Chris Lu
937cfacc01 filer: add "proxyToFileId" to reverse proxy to a volume server 2021-01-23 03:43:48 -08:00
Chris Lu
20ef3bb8d4 mount: use direct_io to avoid OS page cache 
fix https://github.com/chrislusf/seaweedfs/issues/1752
 2021-01-22 22:39:46 -08:00
Chris Lu
711c3f3939 filer: fix http status 206 setting 
fix https://github.com/chrislusf/seaweedfs/issues/1753
 2021-01-22 00:26:30 -08:00
Chris Lu
024c186a48 filer.meta.tail: output json format for easier parsing 2021-01-21 22:39:25 -08:00
Chris Lu
84f05787f8 mount: limit background requests and congestion threshold 2021-01-20 23:30:14 -08:00
Chris Lu
f98817cfe6 filer: support appending to a file 2021-01-20 13:40:32 -08:00
Chris Lu
1b8e3da295 refactoring 2021-01-20 12:59:54 -08:00
Chris Lu
1a3f859c58 filer store: postgres configuration adjust text 2021-01-19 18:12:07 -08:00
Chris Lu
b434f7e4e0 filer store: postgres adjust default config, add optioanl schema 2021-01-19 18:10:36 -08:00
Chris Lu
d5add83e85 filer store: add postgres2 2021-01-19 18:07:29 -08:00
Chris Lu
52a8f1470e filer store: add mysql2 2021-01-19 17:21:50 -08:00
Chris Lu
93b3adba98 fix bucket creation 2021-01-19 15:55:51 -08:00
Chris Lu
fa0c8d5283 fix error 2021-01-19 14:05:48 -08:00
Chris Lu
4c5b752b04 restructuring sql stores 2021-01-19 13:53:16 -08:00
Chris Lu
96354208c5 scaffold: add YugabyteDB 2021-01-19 12:35:14 -08:00
Chris Lu
ca8f793978 refactor: prepare for bucket specific tables 2021-01-19 12:34:58 -08:00
Chris Lu
90ce1eec19 add more input params 2021-01-19 11:32:55 -08:00
Chris Lu
f13c082583 add memsql to comments 2021-01-19 11:31:55 -08:00
Chris Lu
e439b65e38 fix test 2021-01-18 01:48:00 -08:00
Chris Lu
c7197470fd 2.21 2021-01-18 01:25:04 -08:00
Chris Lu
5a3386e39b Revert "Revert "mount: fake support for socket/block/character/fifo devices"" 
This reverts commit 3d5cb7eb86.
 2021-01-18 01:15:07 -08:00
Chris Lu
67faa56920 Revert "Revert "mount: add more entry type"" 
This reverts commit 4085b79d38.
 2021-01-18 01:14:58 -08:00
Chris Lu
7119e7b491 mount: ensure reading from dirty pages 2021-01-18 01:14:42 -08:00
Chris Lu
de876c795d minor fix 2021-01-18 01:14:27 -08:00
Chris Lu
4085b79d38 Revert "mount: add more entry type" 
This reverts commit d7d907be27.
 2021-01-18 00:19:19 -08:00
Chris Lu
3d5cb7eb86 Revert "mount: fake support for socket/block/character/fifo devices" 
This reverts commit 61ef2d8658.
 2021-01-18 00:18:57 -08:00
Chris Lu
61ef2d8658 mount: fake support for socket/block/character/fifo devices 
to pass pjdfstest. A distributed file system can not really support these things anyway.
 2021-01-17 23:51:10 -08:00
Chris Lu
d7d907be27 mount: add more entry type 2021-01-17 23:46:39 -08:00
Chris Lu
389426bbb7 s3: listing may repeat on the edge 2021-01-17 22:59:31 -08:00
Chris Lu
09f49d1c04 refactoring 2021-01-16 19:52:15 -08:00
Chris Lu
a4063a5437 add stream list directory entries 2021-01-15 23:56:24 -08:00
Chris Lu
01dc8a43ba fix rocksdb 2021-01-15 18:31:29 -08:00
Chris Lu
16ad74f477 go fmt 2021-01-14 23:11:27 -08:00
Chris Lu
f002e668de change limit to int64 in case of overflow 2021-01-14 23:10:37 -08:00
Chris Lu
19e52fd414 pass along hasMore 2021-01-14 22:49:29 -08:00
Chris Lu
c0bcf6f6e1 implement sql changes 2021-01-14 22:44:22 -08:00
Chris Lu
5d4568b91f implement elastic changes 2021-01-14 22:42:25 -08:00
Chris Lu
893cbc8482 implement c* changes 2021-01-14 22:38:34 -08:00
Chris Lu
5ef43b9b09 implement etcd changes 2021-01-14 22:35:56 -08:00
Chris Lu
54527f0326 implement hbase changes 2021-01-14 22:34:35 -08:00
Chris Lu
b5ceffe188 implement leveldb changes 2021-01-14 22:33:05 -08:00
Chris Lu
c64bfb0e2e implement mongodb changes 2021-01-14 22:28:51 -08:00
Chris Lu
34a846009d implement rocksdb changes 2021-01-14 22:24:26 -08:00
Chris Lu
d2ece29fff implement redis changes 2021-01-14 22:21:31 -08:00
Chris Lu
9a50dbcda0 chagned api 2021-01-14 22:18:38 -08:00
Chris Lu
e1c7bc66f9 change interface 2021-01-14 22:17:49 -08:00
Chris Lu
4aaa97c789 skip not found error in case of race condition 2021-01-14 19:56:14 -08:00
Chris Lu
d0826c2f62 mount: avoid showing "topics" folder 
fix https://github.com/chrislusf/seaweedfs/issues/1325
 2021-01-14 19:03:15 -08:00
Chris Lu
698f58f7c4 filer, s3: add http status 206 as late as possible 
fix https://github.com/chrislusf/seaweedfs/issues/1746
 2021-01-14 02:59:40 -08:00
Chris Lu
fd1d8a2a09 fix compilation 2021-01-14 00:06:15 -08:00
Chris Lu
ae812c07d6
Merge pull request #1743 from LazyDBA247-Anyvision/master 
MYSQL - Better Connection Pool Control
 2021-01-14 00:04:44 -08:00
Chris Lu
f20743bb38 remove WIP status 2021-01-13 22:21:34 -08:00
LazyDBA247-Anyvision
8eed763b97 better mysql connection pool management 
adding SetConnMaxLifetime configuration (https://golang.org/pkg/database/sql/#DB.SetConnMaxLifetime)
to enable refresh of connections.
 2021-01-14 08:14:21 +02:00
Chris Lu
f17aa1d06c skip directory loop if dropping the bucket 2021-01-13 13:49:04 -08:00
Chris Lu
ca73013453 leveldb3: ensure repeatable adding removing buckets 2021-01-13 13:48:48 -08:00
Chris Lu
e2c7e3fe6d better locks 2021-01-13 13:20:33 -08:00
Chris Lu
819a85f59a avoid viper concurrent access 2021-01-13 13:14:52 -08:00
Chris Lu
3fb2ed9093 filer.meta.tail: optionally submit metadata changes to ElasticSearch 2021-01-13 03:31:37 -08:00
Chris Lu
c5df2577f5 rename 2021-01-13 00:31:19 -08:00
Chris Lu
90c5077610 rename "weed watch" to "weed filer.meta.tail" 2021-01-12 18:48:01 -08:00
Chris Lu
5cd9c9e642 remove println 2021-01-12 18:47:38 -08:00
Chris Lu
def6d8e565 fix test 2021-01-12 12:32:54 -08:00
Chris Lu
1c7faf1100 Revert "close opened file" 
This reverts commit 2880160113.
 2021-01-12 11:07:14 -08:00
Chris Lu
2880160113 close opened file 2021-01-12 11:03:08 -08:00
Chris Lu
0ef43a23a7 go fmt 2021-01-12 02:30:12 -08:00
Chris Lu
629c9962e7 filer: add leveldb3 
support dedicated leveldb instance for each bucket
 2021-01-12 02:29:44 -08:00
Chris Lu
361043e6c1 filer store: leveldb2 fix nil entry error if not found 2021-01-12 02:28:57 -08:00
Chris Lu
cfb9342a15 avoid concurrent map updates to viper 2021-01-12 02:28:13 -08:00
Chris Lu
38d516251e Revert "mount: avoid fatal error" 
This reverts commit a4f0bac9b7.
 2021-01-11 13:50:15 -08:00
Chris Lu
a4f0bac9b7 mount: avoid fatal error 2021-01-11 13:26:59 -08:00
Chris Lu
8a7302ce37 s3: KeyCount is zero 
fix https://github.com/chrislusf/seaweedfs/issues/1741
 2021-01-11 03:50:45 -08:00
Chris Lu
2c7148cd55 redis cluster: do not enable route by latency by default 2021-01-11 02:43:53 -08:00
Chris Lu
acc0d5cd2c Revert "Revert "filer store: redis module upgrade"" 
This reverts commit dd0171e35c.
 2021-01-11 02:30:19 -08:00
Chris Lu
dd0171e35c Revert "filer store: redis module upgrade" 
This reverts commit 40161c9cec.
 2021-01-11 02:09:18 -08:00
Chris Lu
394513f598 filer: ensure seamless meta data updates 2021-01-11 02:08:55 -08:00
Chris Lu
6e12a3a490 refactoring 2021-01-11 02:08:26 -08:00
Chris Lu
62ee484d12 refactoring 2021-01-11 00:03:13 -08:00
Chris Lu
1efb51ba84 filer: change to saveToFilerLimit from cacheToFilerLimit 
short circuit saving small files to volume server
 2021-01-10 23:14:46 -08:00
Chris Lu
f0d3b3bf93 2.20 2021-01-09 22:55:22 -08:00
Chris Lu
d3d3f2fb9b mount: default to 128 concurrent writers 2021-01-09 22:53:37 -08:00
Chris Lu
90b117acf1 update ccache version 2021-01-08 02:17:43 -08:00
Chris Lu
e2e0e30a51 filer: redis default to false for useReadOnly and routeByLatency 2021-01-08 01:12:44 -08:00
Chris Lu
40161c9cec filer store: redis module upgrade 2021-01-08 01:08:44 -08:00
Chris Lu
30efaa66c6 server: add option to disable master 2021-01-07 02:35:47 -08:00
Chris Lu
07f4703bfc
Merge pull request #1738 from qieqieplus/rocksdb 
fix rocksdb crash when list directory
 2021-01-07 02:26:41 -08:00
qieqieplus
0f18592315 alter style 2021-01-07 18:21:48 +08:00
qieqieplus
c7e7431fd3 fix seek lastkey may reach EOF 2021-01-07 17:07:56 +08:00
Chris Lu
e327385644 go fmt 2021-01-06 04:22:00 -08:00
Chris Lu
2b76854641 add "weed filer.cat" to read files directly from volume servers 2021-01-06 04:22:00 -08:00
Chris Lu
95ecf0c72f
Merge pull request #1735 from qieqieplus/rocksdb 
ignore decode error for non-entry data
 2021-01-06 02:22:18 -08:00
qieqieplus
0764fccde7 ignore decode error for non-entry data 2021-01-06 17:42:43 +08:00
Chris Lu
ab57767d4b
Merge pull request #1730 from qieqieplus/rocksdb 
impl: TTL per entry for rocksdb; fix package name
 2021-01-05 13:24:45 -08:00
qieqieplus
b2e50f602f
fix time comparison 2021-01-05 19:36:40 +08:00
qieqieplus
bbae13e415 impl: TTL per entry for rocksdb; fix package name 2021-01-05 17:58:40 +08:00
henry
b42c5b71c1 Seaweed custom header are not visible to Vue or javascript 2021-01-05 13:01:29 +08:00