mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-12-25 18:27:54 +08:00
Code Issues Packages Projects Releases Wiki Activity
3,604 Commits 32 Branches 296 Tags 153 MiB
29a4c3944e
Commit Graph

2124 Commits

Author SHA1 Message Date
Chris Lu
e7e8691720 add some tests 2020-03-26 22:21:52 -07:00
Chris Lu
86398d1efe S3 API: set deletion to recursive 
fix https://github.com/chrislusf/seaweedfs/issues/1251
 2020-03-26 11:05:54 -07:00
Chris Lu
1fef598347 set target Dir object when moving 2020-03-26 10:56:40 -07:00
Chris Lu
9e2e07b238 tweak 2020-03-26 10:56:18 -07:00
Chris Lu
cbce793753 rename: set new name for renamed files 2020-03-26 03:30:23 -07:00
Chris Lu
c763a492f5 refactor 2020-03-26 03:30:02 -07:00
Chris Lu
712720864d error checking for remove 2020-03-26 02:21:30 -07:00
Chris Lu
8d9a76005d add back running output 2020-03-26 02:14:48 -07:00
Chris Lu
dd5b582d05 go fmt 2020-03-26 00:09:01 -07:00
Chris Lu
7f0d87b206 tree structured fs cache 
FsCache for FsNode, wrapping fs.Node
 2020-03-26 00:08:14 -07:00
Chris Lu
2e4fadd10a fix fscache move 2020-03-25 22:19:19 -07:00
Chris Lu
966df2ae27 purging skip EC volumes 2020-03-25 02:41:22 -07:00
Chris Lu
6d87a0c5f5 batch delete orphan data by file keys 2020-03-25 02:21:15 -07:00
Chris Lu
5ebc95b69b refactoring 2020-03-25 02:20:19 -07:00
Chris Lu
d79f0a76b1 format output 2020-03-25 00:56:47 -07:00
Chris Lu
d9b56ed88b shell: volume.fsck add options 2020-03-24 22:38:36 -07:00
Chris Lu
e63a79ade8 better handle lock in case of exception 2020-03-24 18:41:25 -07:00
Chris Lu
4d5554b16f fix compilation on travis 2020-03-24 18:40:52 -07:00
Chris Lu
d83e648c0b auto add prefix "fs." 2020-03-24 02:40:51 -07:00
Chris Lu
a875f67a08 fix pwd printout 2020-03-24 02:40:29 -07:00
Chris Lu
00d3f01876 better output format 2020-03-24 02:34:28 -07:00
Chris Lu
0820935290 clean up 2020-03-24 02:20:37 -07:00
Chris Lu
49440cbd2f shell: add volume.fsck 
volume.fsck can find out how much space is not used by the filer
 2020-03-24 02:18:13 -07:00
Chris Lu
0f10833e0b fix compilation 2020-03-24 00:16:12 -07:00
Chris Lu
3794f2721f fix logic 2020-03-24 00:09:12 -07:00
Chris Lu
443b229ef7 disable chunk file output 2020-03-24 00:08:02 -07:00
Chris Lu
bb9b97e2b4 add comments 2020-03-23 23:07:11 -07:00
Chris Lu
8047ec2f51 shell: fs.meta.save add option to export all fileIds for all files 2020-03-23 22:54:46 -07:00
Chris Lu
782d776d2a refactoring 2020-03-23 22:54:02 -07:00
Chris Lu
38e73463f1 fix du block and byte couting 2020-03-23 21:37:04 -07:00
Chris Lu
b51fa81f0e fix directory checking 2020-03-23 21:36:39 -07:00
Chris Lu
e666aeece2 simplify parsing filer host and port 2020-03-23 21:26:15 -07:00
Chris Lu
d151185b7e shell: desupport filer url in the arguments 2020-03-23 20:46:17 -07:00
Chris Lu
40601953bf comments 2020-03-23 02:54:49 -07:00
Chris Lu
daeb18be30 scaffold: fix master.toml typo 
fix https://github.com/chrislusf/seaweedfs/issues/1247
 2020-03-23 01:45:43 -07:00
Chris Lu
b97768c51c refactoring 2020-03-23 01:30:22 -07:00
Chris Lu
45ee3736aa refactoring 2020-03-23 01:25:38 -07:00
Chris Lu
cddcb5cafe same booleans as before 2020-03-23 01:25:23 -07:00
Chris Lu
f84c288852 purge 2020-03-23 01:14:55 -07:00
Chris Lu
a5aa8be19b Update webdav_server.go 2020-03-23 01:14:42 -07:00
Chris Lu
7f1e3c843d refactoring 2020-03-23 01:14:21 -07:00
Chris Lu
c07bcd5065 refactoring 2020-03-23 00:30:02 -07:00
Chris Lu
654a69ff52 refactoring 2020-03-23 00:06:24 -07:00
Chris Lu
c0f0fdb3ba refactoring 2020-03-23 00:01:34 -07:00
Chris Lu
fbca6b29bd refactoring 2020-03-22 23:52:55 -07:00
Chris Lu
e39e78ea8d remove println 2020-03-22 18:37:12 -07:00
Chris Lu
c16dc57a58 fix max volume count reporting 2020-03-22 18:33:19 -07:00
Chris Lu
d6412f27a0 logging 2020-03-22 18:32:56 -07:00
Chris Lu
35208711e5 logging 2020-03-22 18:32:49 -07:00
Chris Lu
e32999108a add auto configured volume max count help messge 2020-03-22 16:27:15 -07:00
Chris Lu
3137777d83 volume: automatically detect max volume count 2020-03-22 16:21:42 -07:00
Chris Lu
0bf148f49d logging 2020-03-22 16:19:00 -07:00
Chris Lu
bda13ed593 commented prints 2020-03-22 13:09:51 -07:00
Chris Lu
74e23e8d64 FUSE mount: rename correctly across folders 2020-03-22 13:09:33 -07:00
Chris Lu
64000f5c45 FUSE mount: rename under root 
fix https://github.com/chrislusf/seaweedfs/issues/1242
 2020-03-22 13:09:16 -07:00
Chris Lu
9f2c9b666b adjust log level 2020-03-22 01:39:08 -07:00
Chris Lu
ae2ee379c0 consistent 64bit size 2020-03-22 01:37:46 -07:00
Chris Lu
2bdd936fb6 purge code 2020-03-22 01:33:40 -07:00
Chris Lu
e93e986727 webdav: stream read 2020-03-22 01:30:27 -07:00
Chris Lu
65d2ea9fb0 FUSE mount: stream read data with buffer 
fix https://github.com/chrislusf/seaweedfs/issues/1244
 2020-03-22 01:00:36 -07:00
Chris Lu
82bfad5b86 refactoring 2020-03-21 23:48:11 -07:00
Chris Lu
6ff9e2835e filer: skip resizing image if width or height larger than original image 
fix https://github.com/chrislusf/seaweedfs/issues/1239#issuecomment-602140779
 2020-03-21 22:16:00 -07:00
Chris Lu
7c111f7b75 FUSE mount: make "nonempty" optional 
https://github.com/chrislusf/seaweedfs/issues/1094
 2020-03-21 19:14:25 -07:00
Chris Lu
3b3e063f25 1.67 2020-03-20 23:42:49 -07:00
Chris Lu
d3e4a31058 filer: fix where deletion can miss under high concurrency 2020-03-20 23:39:32 -07:00
Chris Lu
cbfe31a9a8 idx file sync before compaction 2020-03-20 23:38:46 -07:00
Chris Lu
3505b06023 report deletion error in the log 2020-03-20 23:38:24 -07:00
Chris Lu
4bc19f1e52 master: fix connections count reporting 2020-03-20 22:41:37 -07:00
Chris Lu
e1b5ac5c04 1.66 2020-03-20 20:39:48 -07:00
Chris Lu
308688c8d0 filer: add back image resizing capability 2020-03-20 20:31:11 -07:00
Chris Lu
f251d03673 refactoring 2020-03-20 15:46:16 -07:00
Chris Lu
11372dd548 refactoring 2020-03-20 15:38:04 -07:00
Chris Lu
c4bea45099 S3 API: fix DeleteMultipleObjectsHandler 
fix https://github.com/chrislusf/seaweedfs/issues/1241
 2020-03-20 14:17:31 -07:00
Chris Lu
165b0d22a4 1.65 2020-03-20 00:05:42 -07:00
Chris Lu
81797a059a volume: sync volume file right before compaction 
fix https://github.com/chrislusf/seaweedfs/issues/1237
 2020-03-19 23:54:52 -07:00
Chris Lu
709f231e23 tiered storage: add s3 endpoint for private s3 implementation 
fix https://github.com/chrislusf/seaweedfs/issues/1238
 2020-03-19 21:13:56 -07:00
Chris Lu
d848d08944 use public url also for deletions 2020-03-18 11:16:45 -07:00
Chris Lu
6566c8e114 weed upload: add usePublicUrl option 
fix https://github.com/chrislusf/seaweedfs/issues/1236
 2020-03-18 10:50:53 -07:00
Chris Lu
db9854fa0a 1.64 2020-03-17 10:58:35 -07:00
Chris Lu
bec6ec7db6 go fmt 2020-03-17 10:01:55 -07:00
Chris Lu
b964bbab3d fix compilation 2020-03-17 10:01:24 -07:00
Chris Lu
c3cb6fa1d7 volume: compaction can cause readonly volumes 
address https://github.com/chrislusf/seaweedfs/issues/1233
 2020-03-17 09:43:57 -07:00
Chris Lu
f9b3750ad1 volume: handle repeated writes response 2020-03-15 04:16:00 -07:00
Chris Lu
22400c6633 consistent gzip logic 
local store gzip same as replicated writes
 2020-03-15 04:15:40 -07:00
Chris Lu
560df51def refactoring 2020-03-15 03:11:26 -07:00
Chris Lu
7edbee6f57 volume: proxy writes to remote volume server, with replication or not 
the panic is triggered by uploading a file to a volume server not holding the designated replica.
2020-03-15 10:20:14.365488 I | http: panic serving 127.0.0.1:57124: runtime error: invalid memory address or nil pointer dereference
goroutine 119 [running]:
net/http.(*conn).serve.func1(0xc0001a8000)
	/home/travis/.gimme/versions/go1.14.linux.amd64/src/net/http/server.go:1772 +0x139
panic(0x2316fe0, 0x3662900)
	/home/travis/.gimme/versions/go1.14.linux.amd64/src/runtime/panic.go:973 +0x396
github.com/chrislusf/seaweedfs/weed/topology.getWritableRemoteReplications(0xc00009c000, 0x2, 0x7ffeefbffbd2, 0xe, 0x0, 0xa, 0x0, 0x0, 0xbb4bf1f7)
	/home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/topology/store_replicate.go:157 +0x53
github.com/chrislusf/seaweedfs/weed/topology.ReplicatedWrite(0x7ffeefbffbd2, 0xe, 0xc00009c000, 0xc000000002, 0xc000472750, 0xc0001b2200, 0x0, 0x1, 0x0)
	/home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/topology/store_replicate.go:29 +0xc7
github.com/chrislusf/seaweedfs/weed/server.(*VolumeServer).PostHandler(0xc0001513f0, 0x292bde0, 0xc0001fe2a0, 0xc0001b2200)
	/home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/server/volume_server_handlers_write.go:52 +0x56f
github.com/chrislusf/seaweedfs/weed/server.(*VolumeServer).privateStoreHandler(0xc0001513f0, 0x292bde0, 0xc0001fe2a0, 0xc0001b2200)
	/home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/server/volume_server_handlers.go:37 +0x21f
net/http.HandlerFunc.ServeHTTP(0xc0004420e0, 0x292bde0, 0xc0001fe2a0, 0xc0001b2200)
	/home/travis/.gimme/versions/go1.14.linux.amd64/src/net/http/server.go:2012 +0x44
net/http.(*ServeMux).ServeHTTP(0xc0001fc800, 0x292bde0, 0xc0001fe2a0, 0xc0001b2200)
	/home/travis/.gimme/versions/go1.14.linux.amd64/src/net/http/server.go:2387 +0x1a5
net/http.serverHandler.ServeHTTP(0xc0001781c0, 0x292bde0, 0xc0001fe2a0, 0xc0001b2200)
	/home/travis/.gimme/versions/go1.14.linux.amd64/src/net/http/server.go:2807 +0xa3
net/http.(*conn).serve(0xc0001a8000, 0x2934420, 0xc000212400)
	/home/travis/.gimme/versions/go1.14.linux.amd64/src/net/http/server.go:1895 +0x86c
created by net/http.(*Server).Serve
	/home/travis/.gimme/versions/go1.14.linux.amd64/src/net/http/server.go:2933 +0x35c
Eg:
server A (datacenter 1) and server B (datacenter 2) hold replica (100) for volume 1.
If you upload a file with a key 1,xxxxx to server C (datacenter 3) will trigger the panic on server C.
The server C should either proxy upload file to the correct volume server or should return an HTTP error code and not panic.
 2020-03-15 02:50:42 -07:00
Chris Lu
7b37178716 filer: close stores if interrupted 2020-03-14 20:30:26 -07:00
Chris Lu
d022b6bc0e fix compilation 2020-03-14 16:32:16 -07:00
Chris Lu
0576a27f44 protect against possible nil 
which is unlikely to happen though
 2020-03-14 16:06:06 -07:00
Chris Lu
cc52e8ca87 1.63 2020-03-14 14:25:07 -07:00
Chris Lu
ee635dcc36 refactoring 2020-03-14 14:08:00 -07:00
Chris Lu
18cda6adbb cleaner fix 2020-03-14 13:55:32 -07:00
Chris Lu
17efa361f9 fix possible nil 2020-03-14 13:53:03 -07:00
Chris Lu
91e0a987e4 1.62 2020-03-14 01:03:59 -07:00
Chris Lu
e2e691d9c2 clean up, add test 2020-03-13 23:53:15 -07:00
Chris Lu
3cc9e85895 volume: vacuum pass preallocate variable 2020-03-13 16:17:44 -07:00
Chris Lu
4ce23e8809 fix compilation error 2020-03-13 15:51:38 -07:00
Chris Lu
c90eb0da1f volume: handling readonly volumes after compaction 
ensure readonly volumes are not added as writable
 2020-03-13 15:41:27 -07:00
Chris Lu
e85da50ad4 remove tikv, move its support to "tikv" branch 2020-03-11 20:59:00 -07:00
Chris Lu
d439d83772 volume: follow compactionBytePerSecond 
related to https://github.com/chrislusf/seaweedfs/issues/1108
 2020-03-11 10:32:17 -07:00
Chris Lu
4237a813cc skip printouts 2020-03-11 01:13:54 -07:00
Chris Lu
ebc739afb5 filer store postgres: skip empty parameters 
better fix https://github.com/chrislusf/seaweedfs/issues/1227
 2020-03-11 01:13:54 -07:00
HongyanShen
81610ed006 fix: #1226 2020-03-11 14:37:14 +08:00
Chris Lu
60f5f05c78 1.61 2020-03-10 13:52:49 -07:00
Chris Lu
7f3ce7b6d6 adjust tikv build env 2020-03-10 13:36:45 -07:00
Chris Lu
d3e1ef64b3 ensure error checking on windows 2020-03-10 13:33:26 -07:00
Chris Lu
88a110e67e release file handle 2020-03-10 13:32:22 -07:00
Chris Lu
43e6261579 volume: fix memory leak 2020-03-10 13:31:23 -07:00
Chris Lu
d214cefc2e filer: list diretory pagination make up for the expired entries 2020-03-09 23:28:01 -07:00
Chris Lu
e6de42f888 1.60 2020-03-09 22:32:40 -07:00
Chris Lu
e73b0c2457 remove not ready status 2020-03-09 22:31:14 -07:00
Chris Lu
0871d2cff0 volume: fix memory leak during compaction 
fix https://github.com/chrislusf/seaweedfs/issues/1222
 2020-03-09 22:29:02 -07:00
Chris Lu
14ae33d642 adjust logging 2020-03-09 10:36:03 -07:00
Chris Lu
89eb05b50f filer: support TTL for all filer stores 2020-03-09 01:02:01 -07:00
Chris Lu
8a899992f2 filer: fix ttl parsing 
fix https://github.com/chrislusf/seaweedfs/issues/1225
 2020-03-09 00:16:10 -07:00
Chris Lu
c32f95c380 1.59 2020-03-08 22:02:34 -07:00
Chris Lu
3934c69757 clean up 2020-03-08 21:54:47 -07:00
Chris Lu
2e3f6ad3a9 filer: remember content is gzipped or not 2020-03-08 21:39:33 -07:00
Chris Lu
5ac6297c68 adjust parameter names 2020-03-08 17:03:37 -07:00
Chris Lu
e04c1a1361 filer: remove chunk manifest file support 
This is not needed for filer
 2020-03-08 17:03:27 -07:00
Chris Lu
1dc30214cb mark encryptVolumeData as not ready 2020-03-08 17:02:47 -07:00
Chris Lu
9b3109a5d8 filer: processing all response headers, no pass through to volume server 
* filer calculate MD5 etag
* filer handle response headers, instread of pass it to volume servers
 2020-03-08 15:42:44 -07:00
Chris Lu
11fceaf2f7 refactoring 2020-03-07 18:09:30 -08:00
Chris Lu
1a75fbfce0 refactoring 2020-03-07 18:06:48 -08:00
Chris Lu
afb20de14c breaks dependency loop 2020-03-07 17:01:39 -08:00
Chris Lu
8645283a7b fuse mount: avoid lookup nil entry 
fix https://github.com/chrislusf/seaweedfs/issues/1221
 2020-03-07 16:51:46 -08:00
Chris Lu
936e7cdbfb pass in filer.copy cipher option 2020-03-07 12:46:00 -08:00
Chris Lu
9228ff192c encryption works 2020-03-07 11:08:57 -08:00
Chris Lu
0375ce2c2e filer: set mime type from volume server 2020-03-07 07:25:15 -08:00
Chris Lu
1ae83c2938 go fmt 2020-03-07 06:12:57 -08:00
Chris Lu
564629444b format 2020-03-07 06:12:20 -08:00
Chris Lu
8c3e25b38a mimee type default "application/octet-stream" to empty string 2020-03-07 06:08:08 -08:00
Chris Lu
dba35404e4 filer: HEAD response add content type 2020-03-07 06:08:08 -08:00
Chris Lu
ea1169dc80 filer cipher: single chunk http POST and PUT and read 2020-03-07 06:08:08 -08:00
Chris Lu
e3b8bf5588
Merge pull request #1219 from song-zhang/master 
schedule new volume by free volume number of nodes
 2020-03-07 05:54:06 -08:00
zhangsong
40f70481cd schedule new volume by free volume number of nodes 2020-03-07 21:41:42 +08:00
Chris Lu
13e215ee5c filer: option to encrypt data on volume server 2020-03-06 00:49:47 -08:00
Chris Lu
31c481e3fc fix typo 2020-03-05 16:00:20 -08:00
Chris Lu
465d18930b filer redis: directory listing follow ttl 
fix https://github.com/chrislusf/seaweedfs/issues/1217
 2020-03-05 10:35:21 -08:00
Chris Lu
e031605248 purge code 2020-03-04 00:44:27 -08:00
Chris Lu
f90c43635d refactoring 2020-03-04 00:39:47 -08:00
Chris Lu
9b7dae8037 filer: fix uploaded chunked file size reporting bug. 2020-03-02 20:27:14 -08:00
Chris Lu
7335e62199 volume: PUT also conditionally gzip compress 2020-03-01 23:37:02 -08:00
Chris Lu
410bce3925 go fmt 2020-03-01 22:39:08 -08:00
Chris Lu
dc40413847 fix error 2020-03-01 22:21:13 -08:00
Chris Lu
6a8484b4ae master able to list all master clients by type 2020-03-01 22:13:47 -08:00
Chris Lu
0ca68a2a6d WIP 2020-03-01 22:13:15 -08:00
Chris Lu
4532640ffd 1.58 2020-03-01 17:59:12 -08:00
Chris Lu
151114ff59 volume: fix readonly status reporting 2020-03-01 17:57:37 -08:00
Chris Lu
1df13d0d6d adjust instructions 2020-02-27 13:15:21 -08:00
Chris Lu
f87f2045b3 delete old file first on windows 
fix https://github.com/chrislusf/seaweedfs/issues/1210
 2020-02-27 00:59:35 -08:00
Chris Lu
757c7d67ad avoid nil DataBackend 2020-02-27 00:07:25 -08:00
Chris Lu
555413d9fc weed queue starts 2020-02-27 00:07:13 -08:00
Chris Lu
430b5a49e2 unused code 2020-02-26 19:35:00 -08:00
Chris Lu
8c1da71402 remove dead code 2020-02-26 17:27:49 -08:00
Chris Lu
ed0acd1722 go fmt 2020-02-26 16:52:57 -08:00
Chris Lu
37a3628b2e grpc add PermitWithoutStream to client and server options 2020-02-26 16:51:38 -08:00
Chris Lu
556dd76303 s3: wait to connect to filer 2020-02-26 16:49:47 -08:00
Chris Lu
0156e2975a mount: add mode to run external to SeaweedFS container cluster 2020-02-26 16:46:01 -08:00
Chris Lu
543cf1c80e fix compilation error 2020-02-25 23:59:30 -08:00
Chris Lu
b1f377f822 mount: fail fast when connecting to filer 
fix https://github.com/chrislusf/seaweedfs/issues/1034
 2020-02-25 23:51:34 -08:00
Chris Lu
36d6595658 remove ctx 2020-02-25 22:38:27 -08:00
Chris Lu
fd9612d66e remove ctx 2020-02-25 22:37:54 -08:00
Chris Lu
86cce3eb58 fix test 2020-02-25 22:29:01 -08:00
Chris Lu
97ab8a1976 remove ctx if possible 2020-02-25 22:23:59 -08:00
Chris Lu
892e726eb9 avoid reusing context object 
fix https://github.com/chrislusf/seaweedfs/issues/1182
 2020-02-25 21:50:12 -08:00
Chris Lu
bd3254b53f adjust logging 2020-02-25 17:24:08 -08:00
Chris Lu
0841bedb15 move filer assign volume grpc errror to response 2020-02-25 17:15:09 -08:00
Chris Lu
bc38b72a20 s3: implemented DeleteMultipleObjects 2020-02-25 14:38:36 -08:00
Chris Lu
35dde56711 refactoring 2020-02-25 12:58:45 -08:00
Chris Lu
986d63cd5e s3: increase list objects or parts limit from 1000 to 10000 
10x of Amazon S3 limits
 2020-02-25 11:45:40 -08:00
Chris Lu
7d10fdf737 fix directory lookup nil 2020-02-25 11:13:06 -08:00
Chris Lu
e86da5a491 minor 2020-02-25 00:42:48 -08:00
Chris Lu
0644d63748 shell: add commands for bucket 2020-02-24 23:30:01 -08:00
Chris Lu
6ab7368ef2 filer: dynamically create bucket under /buckets folder 2020-02-24 22:28:45 -08:00
Chris Lu
5bcb44eda9 filer: move buckets folder to filer.toml since it should not be changed often 2020-02-24 14:42:57 -08:00
Chris Lu
d8dec2323b s3: move buckets folder configuration to filer 2020-02-24 14:34:14 -08:00
Chris Lu
b06b7ca6e6 adjust UI 2020-02-23 22:01:03 -08:00
Yoni Nakache
0d60bb4427
display bug, header is Free but data was Used 2020-02-24 07:47:27 +02:00
Chris Lu
e96f85ed8a make pb files 2020-02-23 18:06:34 -08:00
Chris Lu
346a1cf0b9
Merge pull request #1207 from LazyDBA247-Anyvision/update-stats-and-status-volume-server 
Update stats and status volume server
 2020-02-23 18:05:25 -08:00
Chris Lu
bbb163c5ad add for https://github.com/chrislusf/seaweedfs/pull/1207 2020-02-23 18:04:31 -08:00
LazyDBA247-Anyvision
4ff513d64d staus route: add DiskStatuses for disk in the volume server status 
whem monitoring server, better to know the status of the disks & volumes in a single route.
 2020-02-23 23:27:09 +02:00
LazyDBA247-Anyvision
ab91dbe1d7 fillInDiskStatus: update function to calculate new stats 2020-02-23 23:25:56 +02:00
LazyDBA247-Anyvision
ec8619176a volume server protobuf: add precentFree & percentUsage to the DiskStatus 2020-02-23 23:24:24 +02:00
Yoni Nakache
cd4c9a365b
DiskStats: adding Total & Percent Usage 
making relevant data visible and readable.
 2020-02-23 22:33:47 +02:00
Chris Lu
6ea394b802 1.57 2020-02-22 21:34:49 -08:00
Chris Lu
4ed6b584e2 s3: access control limited by bucket 2020-02-22 21:34:18 -08:00
Chris Lu
e83bfd0a35 adjust log level 2020-02-22 21:23:30 -08:00
Chris Lu
48b7ad5fa8 s3: deny anonymous type 2020-02-22 14:01:04 -08:00
Chris Lu
f55f49970f move volume server stats to grpc 2020-02-21 21:45:03 -08:00
Chris Lu
3a1d017de2 shell: ensure dc and rack does not change for replicated volumes 
fix https://github.com/chrislusf/seaweedfs/issues/1203
 2020-02-21 21:23:25 -08:00
Chris Lu
621cdbdf58 filer: avoid possible timeouts for updates and deletions 2020-02-20 15:44:17 -08:00
Chris Lu
45156cc2fe add test cases for replication=011 2020-02-20 01:21:11 -08:00
Chris Lu
b07089cff2 webdav out of "unstable" 2020-02-18 19:38:59 -08:00
Chris Lu
dbabdd418e add unhandled request auth type 
fix

2020-02-18 11:43:57.396699 I | http: panic serving 172.28.0.43:50658: runtime error: invalid memory address or nil pointer dereference
goroutine 595 [running]:
net/http.(*conn).serve.func1(0xc0001fe3c0)
    /usr/lib/go/src/net/http/server.go:1767 +0x13b
panic(0x55c4e35f3820, 0x55c4e48b3c40)
    /usr/lib/go/src/runtime/panic.go:679 +0x1b6
github.com/chrislusf/seaweedfs/weed/s3api.(*IdentityAccessManagement).authRequest(0xc0004b84e0, 0xc000115900, 0xc0000bb650, 0x1, 0x1, 0x55c4e399d740)
    /go/src/github.com/chrislusf/seaweedfs/weed/s3api/auth_credentials.go:143 +0x11c
github.com/chrislusf/seaweedfs/weed/s3api.(*IdentityAccessManagement).Auth.func1(0x55c4e3994c40, 0xc0007808c0, 0xc000115900)
    /go/src/github.com/chrislusf/seaweedfs/weed/s3api/auth_credentials.go:111 +0x5e
net/http.HandlerFunc.ServeHTTP(0xc0004b87e0, 0x55c4e3994c40, 0xc0007808c0, 0xc000115900)
    /usr/lib/go/src/net/http/server.go:2007 +0x46
github.com/gorilla/mux.(*Router).ServeHTTP(0xc0004ba000, 0x55c4e3994c40, 0xc0007808c0, 0xc000115700)
    /root/go/pkg/mod/github.com/gorilla/mux@v1.7.3/mux.go:212 +0xe4
net/http.serverHandler.ServeHTTP(0xc00011e0e0, 0x55c4e3994c40, 0xc0007808c0, 0xc000115700)
    /usr/lib/go/src/net/http/server.go:2802 +0xa6
net/http.(*conn).serve(0xc0001fe3c0, 0x55c4e399d680, 0xc000894180)
    /usr/lib/go/src/net/http/server.go:1890 +0x877
created by net/http.(*Server).Serve
    /usr/lib/go/src/net/http/server.go:2927 +0x390
 2020-02-18 09:16:04 -08:00
Chris Lu
443a68484a 1.56 2020-02-17 12:54:45 -08:00
Chris Lu
0a57db7026
Merge pull request #1200 from chrislusf/add_s3_acl 
Add s3 ACL
 2020-02-17 12:53:30 -08:00
Chris Lu
4cdde5f569 configuration stores the identity list 2020-02-17 12:31:59 -08:00
Chris Lu
9232d3ac68 ignore error when counting closed connections 2020-02-14 10:28:02 -08:00
Chris Lu
96c1ae8471 refactoring the close http response 2020-02-14 09:46:36 -08:00
Chris Lu
cf5064d702 properly close http response 2020-02-14 09:09:15 -08:00
Chris Lu
9b6296e77a volume: add grpc file read operation 
This is added more for performance benchmarking.
 2020-02-14 00:37:32 -08:00
panyc16
7b3764fd9e make loadExistingVolume() return bool 2020-02-13 15:19:00 +08:00
Chris Lu
62c34454d8 1.55 2020-02-11 23:00:31 -08:00
Chris Lu
83836f4299 filer: recursively batch delete file chunks 
fix https://github.com/chrislusf/seaweedfs/issues/1197
 2020-02-11 22:54:10 -08:00
Chris Lu
2a6db0fd43 volume: return 204 for unchanged file uploads 
fix https://github.com/chrislusf/seaweedfs/issues/1196
 2020-02-11 09:45:02 -08:00
Chris Lu
33b3bd467c Revert "HEAD operation changes to fasthttp" 
This reverts commit 58f126fd27.
 2020-02-10 13:43:53 -08:00
Chris Lu
58f126fd27 HEAD operation changes to fasthttp 2020-02-10 09:13:29 -08:00
Chris Lu
29945fad51 1.54 2020-02-09 18:07:17 -08:00
Chris Lu
b4abe3c081 unused 2020-02-09 18:02:17 -08:00
Chris Lu
f3ce3166ad add streaming v4 2020-02-09 17:42:17 -08:00
Chris Lu
b90ad6f452 add v2 support 2020-02-09 16:02:05 -08:00
Chris Lu
e7b63d50b1 add s3.config to server.go 2020-02-09 14:31:51 -08:00
Chris Lu
9ed364f053 support acl 2020-02-09 14:30:02 -08:00
Chris Lu
8d94564f41 refactor 2020-02-04 21:16:34 -08:00
Chris Lu
c1288e9eb4 volume: sdx generation uses memdb instead of compactMap 
fix https://github.com/chrislusf/seaweedfs/issues/1194
 2020-02-04 21:12:09 -08:00
Chris Lu
08e4702542 sync before closing index file 
merge from 0181f87d91
 2020-02-04 10:38:12 -08:00
Chris Lu
3a35632d58 sync before closing file 
merge changes from 1747fc2d52
 2020-02-04 10:37:14 -08:00
Chris Lu
dc786a63ac master: add configurable volume growth toml setting 2020-02-03 18:15:16 -08:00
Chris Lu
382ff218d3 filer: set file size, streaming chunk file uploading 
fix https://github.com/chrislusf/seaweedfs/issues/1193
 2020-02-03 17:04:06 -08:00
Chris Lu
40ae533fa3 shell: add volume.configure.replication to change replication for a volume 
fix https://github.com/chrislusf/seaweedfs/issues/1192
 2020-02-02 15:37:23 -08:00
Chris Lu
fb19263a71 fix build 2020-01-31 00:59:48 -08:00
Chris Lu
beb0a1599e 1.53 2020-01-31 00:32:09 -08:00
Chris Lu
b2743afaee remove println 2020-01-31 00:21:18 -08:00
Chris Lu
a80ecbfe84 s3: add s3 copy 
fix https://github.com/chrislusf/seaweedfs/issues/1190
 2020-01-31 00:11:12 -08:00
Chris Lu
3b043ead49
Merge pull request #1189 from eryx67/compression-wav 
Support wav files compression
 2020-01-29 22:37:35 -08:00
Chris Lu
abf90ad7b7 remove testing for viper 2020-01-29 21:38:53 -08:00
Chris Lu
20b1fb9ab7 fix test 2020-01-29 21:24:06 -08:00
eryx67
2a87087590 support wav files compression 2020-01-30 10:11:47 +05:00
Chris Lu
8925f3305d adjust example 2020-01-29 09:11:07 -08:00
Chris Lu
d335f04de6 support env variables to overwrite toml file 2020-01-29 09:09:55 -08:00
Chris Lu
27b94cb65b fix wrong url 
fix https://github.com/chrislusf/seaweedfs/issues/1187
 2020-01-28 00:49:47 -08:00
Chris Lu
f724057051 reduce logs 2020-01-27 21:20:41 -08:00
Chris Lu
c8d543ef51 avoid retry delay 2020-01-27 00:58:26 -08:00
Chris Lu
081bc1ea25 mount: able to write to any part of a file 2020-01-27 00:54:52 -08:00
Chris Lu
2f6bb57979 logging 2020-01-27 00:54:21 -08:00
Chris Lu
234f69452b 1.52 2020-01-26 21:04:40 -08:00
Chris Lu
72a64a5cf8 use the same context object in order to retry 2020-01-26 14:42:11 -08:00
Chris Lu
08e4b56a8a mount: able to handle large git clone 2020-01-26 13:01:11 -08:00
Chris Lu
19a05ad174 add test cases 2020-01-26 12:47:23 -08:00
Chris Lu
835da19c09 add logging 2020-01-26 00:50:18 -08:00
Chris Lu
c48fc8b4de grpc send error via response instead of grpc error 2020-01-25 09:17:19 -08:00
Chris Lu
ccf3859152 log 2020-01-25 08:16:43 -08:00
Chris Lu
4433c7bdca logs 2020-01-25 07:34:09 -08:00
Chris Lu
8cccccce9f mount: reuse the entry object for the new directory 
avoid mkdir and then query for the same directory
reduces these "context canceled" issues

attr read directory:"/seaweedfs/other/java/s3copier/src/main/java/com" name:"seaweedfs" : rpc error: code = Canceled desc = context canceled
 2020-01-25 01:15:54 -08:00
Chris Lu
90e30c0170 log 2020-01-25 00:32:18 -08:00
Chris Lu
8a42aa8221 adjust logging 2020-01-25 00:31:53 -08:00
Chris Lu
28d1b34856 logging 2020-01-24 22:13:06 -08:00
Chris Lu
6e25acc681 add logging 2020-01-24 22:01:51 -08:00
Chris Lu
b6e6ca8595 adjust logs 2020-01-24 20:06:58 -08:00
Chris Lu
d4cde5df49 return empty response if not found 2020-01-24 18:07:34 -08:00
Chris Lu
3cdb27fafc adjust logging 2020-01-24 17:55:39 -08:00
Chris Lu
6107509c64 adjust logging 2020-01-24 17:26:18 -08:00
Chris Lu
e6f9f5da3a adjust logging 2020-01-24 17:26:03 -08:00
Chris Lu
24f62ebe76 minor changes 2020-01-24 10:02:53 -08:00
Chris Lu
5f1109f143 reduce log 2020-01-24 01:41:51 -08:00
Chris Lu
9d93c83654 mount: return IO error for mkdir 2020-01-24 01:41:31 -08:00
Chris Lu
107e8a56ea retry context canceled request 2020-01-24 01:40:51 -08:00
Chris Lu
2f75264ec7 mount: avoid use uint64 for greater than zero test 2020-01-23 22:12:57 -08:00
Chris Lu
c936a12afa mount: update the file size so file Attr() can read 
if file Attr() during file writes, the OS, at least for Mac, may try to resend the data to write, which usually are empty.
 2020-01-23 21:59:58 -08:00
Chris Lu
c2e589f202 mount: better combines connected intervals to write to volume servers 2020-01-22 23:00:04 -08:00
Chris Lu
6a5c037099 fix http range requests 2020-01-22 22:59:12 -08:00
Chris Lu
72e1aff237 add lock on dirty pages 2020-01-22 15:38:25 -08:00
Chris Lu
66ded8804a adjust logging 2020-01-22 15:37:59 -08:00
Chris Lu
6b48d246a5 mount: read data that is just written 
able read on data not flushed
multiple file open shares the same file handle

fix https://github.com/chrislusf/seaweedfs/issues/1182 on linux
 2020-01-22 13:42:03 -08:00
Chris Lu
09f4ceef3a mount: reuse previous file node 2020-01-22 11:43:43 -08:00
Chris Lu
d4e75a0d18 filer: option to create file only if it is new, O_EXCL 2020-01-22 11:42:40 -08:00
Chris Lu
9b01a99d9a adjust logging 2020-01-21 22:45:50 -08:00
Chris Lu
c8b2dac6c1 volume: avoid sharing volume dat file handle 
possibly help on https://github.com/chrislusf/seaweedfs/issues/1184
 2020-01-21 21:18:01 -08:00
Chris Lu
bb1be61602 1.51 2020-01-20 20:30:03 -08:00
Chris Lu
fca6152c3a fix print format 2020-01-20 20:24:23 -08:00
Chris Lu
a990ef2106 mount: fix problems found in issue 1182 
fix https://github.com/chrislusf/seaweedfs/issues/1182

always use the non-duplicated fs.Node
Forget() the fs.Node
Rename will also use the right fs.Node
Avoid using the same file handle for the same file
 2020-01-20 20:21:01 -08:00
Chris Lu
630f72f8c5 mount: able to trim a file 2020-01-20 00:00:08 -08:00
Chris Lu
1b0bfbaf59 refactoring 2020-01-19 23:59:46 -08:00
Chris Lu
2f15e93466 mount: rename also clear the cache 
fix https://github.com/chrislusf/seaweedfs/issues/1182
 2020-01-19 12:31:56 -08:00
Chris Lu
04019aa161 mount: deletion ignore old vid 2020-01-19 12:07:26 -08:00
Chris Lu
6a40c688e0 mount: modify file size 2020-01-19 12:07:04 -08:00
Chris Lu
39edcfde4c filer: set inode 
avoid "getcwd: No such file or directory"
 2020-01-19 12:06:19 -08:00
Chris Lu
e9bb0c179f adjust logs 2020-01-18 13:25:17 -08:00
Chris Lu
061a61e018 logs 2020-01-15 19:09:00 -08:00
Chris Lu
230a0e4a33 filer: no entry if not found 2020-01-15 19:08:54 -08:00
Chris Lu
e4b660699f correct comments 
fix https://github.com/chrislusf/seaweedfs/issues/1181
 2020-01-12 20:31:33 -08:00
Chris Lu
d1ab16b6e3 treat it as a single node cluster if empty raft server name 
possible fix for https://github.com/chrislusf/seaweedfs/issues/1118
 2020-01-10 00:37:44 -08:00
Chris Lu
30c7148020 1.50 2020-01-08 22:08:35 -08:00
Chris Lu
8dd260623e protect against nil deletion results 
fix https://github.com/chrislusf/seaweedfs/issues/1180
 2020-01-08 21:50:59 -08:00
Chris Lu
89e16bd2e8 skip error when draining reader 
fix https://github.com/chrislusf/seaweedfs/issues/1179
 2020-01-08 18:07:07 -08:00
Chris Lu
66ab09aa4a 1.49 2020-01-08 09:51:35 -08:00
Chris Lu
f8a20ef35e add the old way to compact as a comment 2020-01-08 09:45:42 -08:00
Chris Lu
288baf37fd saving .vif files correctly 2020-01-08 09:45:26 -08:00
Chris Lu
acf7ca7b93 volume: fix compaction 2020-01-08 09:45:03 -08:00
Chris Lu
943f4986ef fix possible nil 2020-01-08 08:49:18 -08:00
Chris Lu
9995d3bcb5 remove println 2020-01-07 21:49:28 -08:00
Chris Lu
dc9457fb87 fix compilation error 2020-01-07 09:03:47 -08:00
Chris Lu
aa73364a24 weed fix: add back .idx file generation 2020-01-06 16:29:59 -08:00
Chris Lu
4e731f1c8b volume: copy volumes also include .vif file 2020-01-04 11:28:29 -08:00
Chris Lu
df636e4c14 remove .vif files 2020-01-03 12:46:42 -08:00
Chris Lu
3eafec4b29 volume: add option to limit file size 2020-01-03 00:37:26 -08:00
Chris Lu
9c139e0601 update help message 2020-01-01 12:38:29 -08:00
Chris Lu
88a80ab557 fix help message 2020-01-01 12:37:38 -08:00
Chris Lu
aa1807e082 1.48 2020-01-01 09:27:02 -08:00
Chris Lu
86d4b18eb5 filer: optional recursive deletion 
fix https://github.com/chrislusf/seaweedfs/issues/1176
 2019-12-31 11:52:54 -08:00
Chris Lu
09043c8e5a fs: synchronized meta file writing 
fix https://github.com/chrislusf/seaweedfs/issues/1175
 2019-12-29 20:19:51 -08:00
Chris Lu
509f314350 keep .vif file for ec to normal volume conversion 2019-12-28 21:52:55 -08:00
Chris Lu
672868b460 always create .vif file 2019-12-28 21:52:06 -08:00
Chris Lu
03ab0b1b9a correct .vif file creation 2019-12-28 21:40:12 -08:00
Chris Lu
b6f7033478 go fmt 2019-12-28 21:37:29 -08:00
Chris Lu
9134092a8d endure creating .vif files 2019-12-28 21:36:15 -08:00
Chris Lu
0cc8347a3a go fmt 2019-12-28 21:13:10 -08:00
Chris Lu
8fbc0a9163 fix edge cases 2019-12-28 12:59:31 -08:00
Chris Lu
37b64a50b4 ec: generate and copy .vif file 2019-12-28 12:44:59 -08:00
Chris Lu
c06f7eb48a load volume info from .vif file, use superblock as a backup 2019-12-28 12:28:58 -08:00
Chris Lu
f4a74e03d1 refactoring: separating .vif from tier file loading 2019-12-28 11:35:27 -08:00
Chris Lu
2000284435 rename volume tier info to volume info 2019-12-28 11:21:49 -08:00
Chris Lu
e8b357fd53 rename from .tier to .vif 2019-12-28 11:17:39 -08:00
Chris Lu
b3c83f9404 fix spelling 2019-12-28 11:16:10 -08:00
Chris Lu
1fd8926ac7 ignore draining error 2019-12-25 21:39:33 -08:00
Chris Lu
48d28d3eb2 tier: support remote file attributes and remember the file extension 2019-12-25 21:37:24 -08:00
divinerapier
4cbb6fa199
feat: drains http body if buffer is too small 
Signed-off-by: divinerapier <poriter.coco@gmail.com>
 2019-12-26 11:28:43 +08:00
divinerapier
84640d07b7
fix: handle errors for ReadUrl 
Signed-off-by: divinerapier <poriter.coco@gmail.com>
 2019-12-26 10:07:07 +08:00
Chris Lu
1346437d71
Merge pull request #1171 from stlpmo-jn/redundant_type_conversion 
remove the redundant type conversion
 2019-12-25 17:02:05 -08:00
Chris Lu
35393b4a02 cleanly destroy remote files 2019-12-25 16:17:58 -08:00
Chris Lu
eecad6062a go fmt 2019-12-25 09:56:42 -08:00
Chris Lu
d960b3474a tier storage: support downloading the remote dat files 2019-12-25 09:53:13 -08:00
stlpmo
f3de4b6c18 remove the redundant type conversion 2019-12-25 10:13:45 +08:00
Chris Lu
3ebeae0c0b ec encode distribute ec data and parity shards evenly 2019-12-24 16:52:21 -08:00
Chris Lu
9ff72f616a go fmt 2019-12-24 14:56:16 -08:00
Chris Lu
efd2f50ede compaction changed to .idx based deletion 2019-12-24 14:55:50 -08:00
Chris Lu
7ba6be2312 volume ui add readonly status 2019-12-24 14:55:26 -08:00
Chris Lu
d8b39fe92a testing 2019-12-24 11:29:26 -08:00
Chris Lu
f42b5bd0f5 rename .sdb to .sdx 2019-12-24 10:19:12 -08:00
Chris Lu
abffe857a1 change btree map to in memory level db 2019-12-24 10:18:56 -08:00
Chris Lu
72a561ab7c refactoring 2019-12-24 08:13:01 -08:00
Chris Lu
9114f7f432
Merge pull request #1169 from stlpmo-jn/http_304_has_no_resp_body 
remove the http 304 response body
 2019-12-24 07:08:46 -08:00
Chris Lu
409a3fe41f
Merge pull request #1168 from stlpmo-jn/optimize_DeleteCollectionFromDiskLocation 
decouple the volume.Destroy() from the operation of unmountVolume()
 2019-12-24 07:08:04 -08:00
Chris Lu
37a836c8a7
Merge pull request #1167 from stlpmo-jn/volume_server_core_dump 
resolve issue : create volume on no free space disk, the volume server core dump
 2019-12-24 07:07:41 -08:00
stlpmo
f87ccbddb4 remove the http 304 response body 2019-12-24 17:56:50 +08:00
stlpmo
38e4b79125 decouple the volume.Destroy() from the operation of unmountVolume() 2019-12-24 17:20:34 +08:00
stlpmo
f300c35266 resolve issue : when the host disk full, the volume server will core dump 2019-12-24 16:36:15 +08:00
Chris Lu
1ad34a2487 ed.decode prefers servers with most data shards 2019-12-24 00:00:45 -08:00
Chris Lu
a18f62bbe7 only copy required shards 2019-12-23 18:06:13 -08:00
Chris Lu
c7d5a0b00c adjust help message 2019-12-23 17:58:47 -08:00
Chris Lu
8a2d529759 go fmt 2019-12-23 12:48:53 -08:00
Chris Lu
09ca936c78 shell: add ec.decode command 2019-12-23 12:48:20 -08:00
Chris Lu
dda5c6d3cb fmt 2019-12-22 04:31:36 -08:00
Chris Lu
a9b1ee18af default redis reads options 
details in https://github.com/chrislusf/seaweedfs/pull/1165
 2019-12-21 09:30:51 -08:00
Chris Lu
fcbd04c425 default to false in scaffold.go 2019-12-21 08:31:27 -08:00
Chris Lu
62b1a346ef set default 2019-12-20 20:57:38 -08:00
Chris Lu
cd47112c40
Merge pull request #1165 from dqsully/master 
Add readOnly and routeByLatency options to Redis Cluster Store
 2019-12-20 20:56:34 -08:00
Chris Lu
573b74f24f add SetDefault function for Configuration 2019-12-20 20:56:14 -08:00