Commit Graph

705 Commits

Author SHA1 Message Date
Chris Lu
0d2ec832e2 rename from volumeType to diskType 2020-12-13 11:59:32 -08:00
Chris Lu
d156c74ec0 volume server set volume type and heartbeat to the master 2020-12-13 03:11:24 -08:00
Chris Lu
e9cd798bd3 adding volume type 2020-12-13 00:58:58 -08:00
Chris Lu
06bb7bf6c0 adding /usr/local/etc/seaweedfs for bsd style config search path
related to https://reviews.freebsd.org/D27391
2020-12-07 16:46:48 -08:00
Chris Lu
263eb29e9f filer: add option to cache small files to filer store 2020-12-04 22:39:43 -08:00
Chris Lu
6d30b21b10 volume: add "-dir.idx" option for separate index storage
fix https://github.com/chrislusf/seaweedfs/issues/1265
2020-11-27 03:17:10 -08:00
Chris Lu
983fd90010 marked as WIP 2020-11-26 15:08:52 -08:00
Chris Lu
b11449f955 pass in dir.idx parameter to DiskLocation 2020-11-26 14:59:03 -08:00
Chris Lu
ef3e068c51 deprecating "filer.options.buckets_fsync" in filer.toml 2020-11-16 16:56:49 -08:00
Chris Lu
827930e249
Merge pull request #1610 from kmlebedev/prefer_read_in_this_dc
prefer to read from volumes in this data center
2020-11-11 22:09:41 -08:00
Konstantin Lebedev
0880aff224 update dc flag desc 2020-11-12 10:07:52 +05:00
Konstantin Lebedev
fc7baef5bb fiil serverUrls sorted by data center 2020-11-12 02:13:33 +05:00
Chris Lu
745ee8d8f3 avoid unnecessary error message
fix https://github.com/chrislusf/seaweedfs/issues/1611
2020-11-11 13:00:05 -08:00
Chris Lu
8750cac090 move to util.RetryWaitTime 2020-11-01 02:36:43 -08:00
Chris Lu
de86945aeb go fmt 2020-10-31 16:45:38 -07:00
Chris Lu
8826601be1 mount: optional limit for the number of concurrent writers 2020-10-30 21:22:20 -07:00
Chris Lu
7ecbb4b3c9 clean up 2020-10-30 10:46:31 -07:00
Kenny
9cc6575ab5
Update filer_copy.go
修改 weed filer.copy 上传目录时文件名变子目录的问题
2020-10-30 18:16:49 +08:00
Chris Lu
47047516d6 Revert "increase default volume file size limit to 1024"
This reverts commit 0983060a

I am confused why this was changed to 1024 by myself. 1GB should be too large for most cases.
2020-10-29 15:46:29 -07:00
Chris Lu
c6d9974fe7 server: Add option to enable/disable volume server
fix https://github.com/chrislusf/seaweedfs/issues/1572
2020-10-29 00:24:18 -07:00
Chris Lu
e219c57849 passing full path when assign volume locations 2020-10-25 15:46:29 -07:00
Chris Lu
f375b93aef renaming 2020-10-25 15:32:43 -07:00
Chris Lu
63b0fb54f7 minor 2020-10-25 14:15:53 -07:00
Chris Lu
147d4d23b6 filer.copy: automatically use the right bucket 2020-10-25 11:21:33 -07:00
Chris Lu
da36abf033 go fmt 2020-10-24 20:12:04 -07:00
Chris Lu
b7b1eb51b1 default to 2 for better performance in common computer hardware 2020-10-24 20:08:26 -07:00
Chris Lu
1b17f71939 adjust election timeout to 10 seconds 2020-10-23 23:06:44 -07:00
Chris Lu
0cea84e7ab make it easy with setting up multiple masters in a clcuster 2020-10-23 18:18:46 -07:00
Konstantin Lebedev
a9a7005687 set desc of option 2020-10-22 11:23:00 +05:00
Chris Lu
6c92fa349d filer: support default rack
fix https://github.com/chrislusf/seaweedfs/issues/1546
2020-10-20 17:41:39 -07:00
Chris Lu
ee1fc6558a refactor 2020-10-16 11:02:44 -07:00
Chris Lu
3f7d1d1bf1 Only wait on retryable requests 2020-10-13 00:29:46 -07:00
Chris Lu
7704469d29 mount: outsideContainerClusterMode changed to use volume server publicUrl 2020-10-11 20:42:15 -07:00
Chris Lu
d155f907c2 mount: configurable read wait time 2020-10-10 20:09:43 -07:00
Chris Lu
78859e35fc s3: adjust config examples 2020-10-08 10:12:09 -07:00
Chris Lu
a8624c2e4f read from alternative replica
related to https://github.com/chrislusf/seaweedfs/issues/1512
2020-10-07 22:49:04 -07:00
Chris Lu
da4edf3651 master: check peers for existing leader before starting a leader election
fix https://github.com/chrislusf/seaweedfs/issues/1509
2020-10-07 01:25:39 -07:00
Chris Lu
c543762e23 volume: auto expands -max setting
I have tried to run weed volume but got errors...
cmd:
weed volume -max=0 -mserver="127.0.0.1:9333" -port=8080 -dir="/var/www/d1/sfs,/var/www/d2/sfs,/var/www/d3/sfs"
log:
I1006 15:09:26 17040 file_util.go:23] Folder /var/www/d1/sfs Permission: -rwxr-xr-x
I1006 15:09:26 17040 file_util.go:23] Folder /var/www/d2/sfs Permission: -rwxr-xr-x
I1006 15:09:26 17040 file_util.go:23] Folder /var/www/d3/sfs Permission: -rwxr-xr-x
F1006 15:09:26 17040 volume.go:142] 3 directories by -dir, but only 1 max is set by -max
What am I doing wrong? Thx.
2020-10-06 09:05:30 -07:00
Chris Lu
2c20ef72ae fix starting problem 2020-10-03 13:56:38 -07:00
Chris Lu
d77e3c96e9 FUSE: use the mount directory permissions 2020-10-03 13:37:35 -07:00
Устюжанин Антон Александрович
6fbfe73683 fix: resumeState 2020-10-03 22:16:47 +05:00
Устюжанин Антон Александрович
dc31b19469 fix: restore raft state 2020-10-03 14:03:41 +05:00
Устюжанин Антон Александрович
8c82fb7e5f fix: restore raft state 2020-10-02 23:01:20 +05:00
Chris Lu
79ab10e300 adjust help message 2020-09-30 09:32:00 -07:00
Chris Lu
55cb68c09c filer: replication follows master config if not specified 2020-09-30 09:15:55 -07:00
Chris Lu
e6552b5e1e filer: able to start s3 together 2020-09-27 23:00:43 -07:00
Chris Lu
8516517c48 filer store: Cassandra supports username/password 2020-09-24 21:31:06 -07:00
Chris Lu
043b063136
Merge pull request #1482 from hilimd/master
Fix: s3 delete object
2020-09-24 18:21:34 -07:00
limd
48c578410f mount: rollback default value 2020-09-25 09:18:52 +08:00
Chris Lu
4856bce0ee adjust for metrics port 2020-09-24 10:21:23 -07:00
Chris Lu
d2d3aec3e1 consolidate to one metricsPort in "weed server" mode 2020-09-24 09:55:02 -07:00
Chris Lu
2e9099369e
Merge pull request #1489 from kmlebedev/promhttp
Promhttp
2020-09-24 09:50:48 -07:00
Konstantin Lebedev
98e9de6e11 fix style 2020-09-24 17:48:39 +05:00
Konstantin Lebedev
324e44d4b3 add start metrics server 2020-09-24 17:45:39 +05:00
limd
cbca14edc5 mount: fix k8s pvc and os mount directory permission bug 2020-09-24 18:07:16 +08:00
limd
59e91e9c7e mount: fix k8s pvc mount directory permission 2020-09-24 10:25:45 +08:00
limd
9b21ec27e4 mount: auto created dir set corrct umask fix bug 2020-09-23 18:35:37 +08:00
Chris Lu
f7a0ccb595 mount: auto created dir set corrct umask 2020-09-23 02:31:19 -07:00
James Hartig
658fc2e5b6 Allow option to enable volume pprof on server 2020-09-21 22:43:10 -04:00
Chris Lu
d013d09a9b adjust logging 2020-09-20 15:38:59 -07:00
Chris Lu
2c21eb1971 volume: get metrics configuration from master
fix https://github.com/chrislusf/seaweedfs/issues/1354
2020-09-19 00:03:00 -07:00
Chris Lu
6544e60bea s3 add metrics
empty for now
2020-09-17 06:56:15 -07:00
Chris Lu
feca07bf96 Filer: ip bind
fix https://github.com/chrislusf/seaweedfs/issues/1470
2020-09-16 10:26:08 -07:00
Chris Lu
684a875876 update help message 2020-09-15 00:40:38 -07:00
Chris Lu
103fafe00b weed export: print out [start,stop) content range 2020-09-14 22:57:23 -07:00
Chris Lu
5d6753fb98 shell: add volumeServer.leave command 2020-09-13 21:25:51 -07:00
Chris Lu
1af95c5b76 refactoring 2020-09-13 12:41:26 -07:00
Chris Lu
1a7afe7e6a adjust default value
related to https://github.com/chrislusf/seaweedfs/issues/1453
2020-09-12 12:46:42 -07:00
Chris Lu
f2723c1bc8 do not idx file format
revert c9ab8d05fa
2020-09-12 12:42:36 -07:00
Chris Lu
446e476a11 go fmt 2020-09-12 04:08:03 -07:00
Chris Lu
cd9b89ba55 reduce default wait time to 10s before shutting down 2020-09-12 04:05:33 -07:00
Chris Lu
1b8094ef75 weed export: export deleted files 2020-09-11 02:05:14 -07:00
Chris Lu
c9ab8d05fa fixes for reading deleted fid 2020-09-10 14:42:52 -07:00
Chris Lu
0d9b858cfb
Merge pull request #1455 from taozix/master
add more basic elastic options.
2020-09-10 01:31:17 -07:00
ruitao.liu
719dc43af1 modify elastic urls from string to array. 2020-09-10 16:24:09 +08:00
ruitao.liu
3f7fbfddca add more basic elastic options. 2020-09-10 14:22:07 +08:00
Chris Lu
47b3f932e4 watch: adjust output format 2020-09-09 22:34:48 -07:00
Chris Lu
7f69acd1f2 sync pprof 2020-09-09 11:33:52 -07:00
Chris Lu
387ab6796f filer: cross cluster synchronization 2020-09-09 11:21:23 -07:00
Chris Lu
4fc0bd1a81 return http response directly 2020-09-09 03:53:09 -07:00
ruitao.liu
83080b5e03 ES backended filer support kv ops. 2020-09-04 15:40:13 +08:00
Chris Lu
46f65a84a6 filer: elastic7 adjust default value 2020-09-03 20:12:38 -07:00
ruitao.liu
a93d27d1e8 new filer option to es v7. 2020-09-03 16:34:58 +08:00
Chris Lu
7e1aad0b54 mount: map uid/gid between local and filer 2020-09-03 00:07:22 -07:00
Chris Lu
63ad1abcce watch: follow changes for a pattern 2020-08-28 19:43:04 -07:00
Chris Lu
6bc618051c log message 2020-08-26 09:32:56 -07:00
Chris Lu
9dae4e9d3e fix logs 2020-08-26 09:31:11 -07:00
Chris Lu
4c1a3187bf fix log error 2020-08-26 09:25:54 -07:00
Konstantin Lebedev
464d4c82ec stop send heartbeat before stop volume server 2020-08-26 16:16:11 +05:00
Chris Lu
fe01191b5b support read option readDeleted=true 2020-08-18 19:22:16 -07:00
Chris Lu
6a92f0bc7a refactoring to typed Size
Go is amazing with refactoring!
2020-08-18 17:04:28 -07:00
Chris Lu
0983060a90 increase default volume file size limit to 1024
avoid possible large chunk size set on mount or filer
2020-08-13 09:07:22 -07:00
Chris Lu
eed525b717 FUSE mount: remove DirListCacheLimit
outdated parameter
2020-07-28 09:24:39 -07:00
Chris Lu
a566bfc6e1 s3: use bucket in the domain
fix https://github.com/chrislusf/seaweedfs/issues/1405
2020-07-28 08:47:27 -07:00
Chris Lu
f43146b237 resolve directories if containing home directory 2020-07-16 22:50:14 -07:00
Chris Lu
befb396892 mount: resolve home directory
fix https://github.com/chrislusf/seaweedfs/issues/1391
2020-07-16 22:38:55 -07:00
Chris Lu
2b2ca7d1dc stop if the directory can not be found
related to https://github.com/chrislusf/seaweedfs/issues/1391
2020-07-16 13:24:53 -07:00
Chris Lu
ddec7b2bb9 go fmt 2020-07-14 07:34:16 -07:00
李远军
6608cb5f43 Revert "remove fixJpgOrientation"
This reverts commit de5ca9b2
2020-07-10 10:08:36 +08:00
Chris Lu
a2eb680f34 filer: default filer store directory
1. set default filer store directory
2. set peers, avoiding empty string counted as 1.
2020-07-07 23:06:48 -07:00
Chris Lu
d198e8c6d6 filer: support cross filer meta data sync if sharing the same store 2020-07-06 09:24:54 -07:00
Chris Lu
59152334c7 default to 1% 2020-07-03 16:42:53 -07:00
Chris Lu
acf4a9ea99 volume: min free space default to 1% 2020-07-03 16:42:02 -07:00
Chris Lu
4d1484628a refactoring 2020-07-03 16:41:30 -07:00
Chris Lu
12f6b683fc default to 8 volumes
in case all the volumes are allocated for a collection
2020-06-28 14:34:21 -07:00
Chris Lu
9033a7d369 removed async option 2020-06-28 10:18:32 -07:00
Chris Lu
1bb8cae65d reverting and working 2020-06-28 10:14:17 -07:00
Chris Lu
98493e0152 volume: add default minFreeSpacePercent
fix https://github.com/chrislusf/seaweedfs/issues/1366
2020-06-21 08:44:06 -07:00
Chris Lu
e7c36f740a refactoring 2020-06-21 08:38:00 -07:00
Chris Lu
8b4a32e782 adjust help message 2020-06-21 08:34:24 -07:00
Chris Lu
b4ec04d464 fix the help message 2020-06-21 08:31:20 -07:00
Chris Lu
de5ca9b258 remove fixJpgOrientation 2020-06-20 12:50:40 -07:00
Chris Lu
ca3516ac6d adjust protoc 2020-06-20 08:00:25 -07:00
Chris Lu
e912fd15e3 renaming 2020-06-19 22:45:27 -07:00
Chris Lu
b78a2dc59b adjust help message 2020-06-19 12:55:09 -07:00
Chris Lu
48880edc2c volume: fix typo in minFreeSpacePercent option 2020-06-14 20:37:05 -07:00
Chris Lu
a0464fc267 fix help
fix https://github.com/chrislusf/seaweedfs/issues/1359
2020-06-12 22:06:21 -07:00
Chris Lu
0d1d83a5a5 mount: add option to auto create dir to mount 2020-06-12 11:46:58 -07:00
Chris Lu
628b27ef3b purge old cache implementation 2020-06-11 01:50:00 -07:00
Chris Lu
b9365de47b fix volume server pprof 2020-06-10 13:10:10 -07:00
Kimbsen
03e8f976d9 added --pprof to volume servers 2020-06-10 11:52:24 +02:00
Chris Lu
45a9d852d3 go fmt 2020-06-05 15:27:10 -07:00
Evgenii Kozlov
2410ffbd79 Name fix 2020-06-05 19:00:38 +03:00
Evgenii Kozlov
0e0db70f55 Set volumes ReadOnly if low free disk space 2020-06-05 18:18:15 +03:00
Chris Lu
393ab07c7c disable pulse seconds
there are a few other hard coded timing settings, which will cause disconnected volume servers.
2020-06-04 10:52:01 -07:00
Chris Lu
bc2ec6774d inject git version into build 2020-06-02 00:10:38 -07:00
Chris Lu
ac48c899e8 add printouts for synchronizing metadata which can be slow for large folders 2020-05-18 10:31:12 -07:00
Chris Lu
0c856e8387 avoid possible wrong arguments
fix https://github.com/chrislusf/seaweedfs/issues/1320
2020-05-18 10:07:12 -07:00
Chris Lu
2bfd810912 combine msgBroker into weed server command 2020-05-17 22:44:20 -07:00
Chris Lu
39e72fb23c go fmt 2020-05-10 03:50:30 -07:00
Chris Lu
1e3e4b3072 add broker connects to filer 2020-05-05 02:05:28 -07:00
Chris Lu
78422c2f69
Merge pull request #1298 from song-zhang/master
add fsync option for benchmark
2020-04-30 20:16:42 -07:00
Chris Lu
871efa4fc1 refactoring
some previous chunk etag was using md5, which should be wrong.
2020-04-30 17:20:47 -07:00
zhangsong
d819865fe7 add fsync option for benchmark 2020-04-30 23:31:08 +08:00
Chris Lu
eab4c9219c randomize benchmark content
fix https://github.com/chrislusf/seaweedfs/issues/1294
2020-04-28 01:50:56 -07:00
Chris Lu
5c57297bd1 metadata log: read from any timestamp 2020-04-28 00:05:57 -07:00
wuyuxiang
6850d28d6b refacotr: move signal handling and pprof to grace package 2020-04-28 14:10:23 +08:00
Chris Lu
0c2248f83a add lock/unlock to example admin scripts 2020-04-23 18:41:48 -07:00
Chris Lu
913aa0f14e mount: default to enable asyncMetaDataCaching 2020-04-23 14:14:28 -07:00
Chris Lu
b8e4238ad2 add placeholder for cached meta data reading 2020-04-21 18:02:08 -07:00
Chris Lu
57df14f76f report error if copying failed 2020-04-21 14:28:46 -07:00
Chris Lu
cf54d70311 filer: add -ip.bind option 2020-04-21 14:21:06 -07:00
Chris Lu
f373232227
Merge pull request #1280 from bukton123/filer_mongodb
Add new filer support mongodb
2020-04-19 20:35:19 -07:00
bukton
3defa718a1 add default config filer mongodb to scaffold.go 2020-04-19 17:52:38 +07:00
Chris Lu
71ffb98475 broker: add profiling 2020-04-19 03:03:40 -07:00
Chris Lu
767f14dfcd master, volume default to detected host 2020-04-18 15:20:20 -07:00
Chris Lu
076c8bd3bc filer master start up with default ip address instead of just localhost 2020-04-18 15:17:27 -07:00
Chris Lu
11f5a6d913 messaging can compile now 2020-04-18 01:12:01 -07:00
Chris Lu
826f96b6f3 fix unreachable code 2020-04-17 10:04:28 -07:00
Chris Lu
2a45897237 broker: read cipher value from filer 2020-04-17 02:29:00 -07:00
Chris Lu
f5a748d33c refactoring 2020-04-16 02:55:09 -07:00
Chris Lu
ce4b369be2 scaffolding messaging 2020-04-16 02:21:23 -07:00
Chris Lu
d8f5985e5e add logs 2020-04-13 12:58:45 -07:00
Chris Lu
7764e0465c refactoring 2020-04-12 21:00:55 -07:00
Chris Lu
6f948e4887 remove configurable topics folder location 2020-04-12 13:07:59 -07:00
Chris Lu
b5a713be68 filer store: add redis2 redis_cluster2 as default redis store 2020-04-12 02:50:41 -07:00
Chris Lu
211d87cf4c mount: option to disable caching 2020-04-12 00:52:54 -07:00
Chris Lu
1c65656fb4 s3: add option to fsync buckets 2020-04-11 23:37:10 -07:00
Chris Lu
5987810e5e set default chunk size to 16 2020-04-11 21:22:52 -07:00
Chris Lu
df97da25f9 mount: add on disk caching 2020-04-11 21:12:41 -07:00
Chris Lu
dce2702e1b fix the instruction 2020-04-10 14:50:10 -07:00
Chris Lu
8db2120bee add logs 2020-04-09 23:42:59 -07:00
Chris Lu
59f40e2027 volume: best effort to detect ip address
fix https://github.com/chrislusf/seaweedfs/issues/1264
2020-04-09 00:26:24 -07:00
Chris Lu
f6a7e79dc3 weed shell: simplify CLI option for filer 2020-04-08 23:57:15 -07:00
Chris Lu
2329d9e0c1 add volume.fix.replication to default scaffold 2020-04-08 23:36:22 -07:00
Chris Lu
4c498e73de filer replication: add s3 endpoint 2020-04-07 17:49:00 -07:00
Chris Lu
2d43f85577 watch entries with common path prefix 2020-04-05 12:51:21 -07:00
Chris Lu
af1f64d244 change from 'weed tail' to 'weed watch' 2020-04-05 12:39:20 -07:00
Chris Lu
bf270d9e8c filer: able to tail meta data changes 2020-04-05 00:51:16 -07:00
Chris Lu
d2f8e50459 make error a little bit more clearer
fix https://github.com/chrislusf/seaweedfs/issues/1259
2020-04-03 00:24:09 -07:00
Chris Lu
5f881d2fd5 mount: fail early rather than after mounted the directory
fix https://github.com/chrislusf/seaweedfs/issues/1258
2020-04-03 00:21:03 -07:00
James Hartig
eae3f27c80 Added treat_replication_as_minimums master toml option 2020-04-01 19:08:48 -04:00
Chris Lu
a75d50bbb8 FUSE: add configurable in memory chunk cache size 2020-03-28 14:07:16 -07:00
Chris Lu
eedd33dda3 refactoring 2020-03-28 13:41:58 -07:00
Chris Lu
40601953bf comments 2020-03-23 02:54:49 -07:00
Chris Lu
daeb18be30 scaffold: fix master.toml typo
fix https://github.com/chrislusf/seaweedfs/issues/1247
2020-03-23 01:45:43 -07:00
Chris Lu
e32999108a add auto configured volume max count help messge 2020-03-22 16:27:15 -07:00
Chris Lu
7c111f7b75 FUSE mount: make "nonempty" optional
https://github.com/chrislusf/seaweedfs/issues/1094
2020-03-21 19:14:25 -07:00
Chris Lu
709f231e23 tiered storage: add s3 endpoint for private s3 implementation
fix https://github.com/chrislusf/seaweedfs/issues/1238
2020-03-19 21:13:56 -07:00
Chris Lu
d848d08944 use public url also for deletions 2020-03-18 11:16:45 -07:00
Chris Lu
6566c8e114 weed upload: add usePublicUrl option
fix https://github.com/chrislusf/seaweedfs/issues/1236
2020-03-18 10:50:53 -07:00
Chris Lu
e85da50ad4 remove tikv, move its support to "tikv" branch 2020-03-11 20:59:00 -07:00
Chris Lu
d439d83772 volume: follow compactionBytePerSecond
related to https://github.com/chrislusf/seaweedfs/issues/1108
2020-03-11 10:32:17 -07:00
Chris Lu
e73b0c2457 remove not ready status 2020-03-09 22:31:14 -07:00
Chris Lu
0871d2cff0 volume: fix memory leak during compaction
fix https://github.com/chrislusf/seaweedfs/issues/1222
2020-03-09 22:29:02 -07:00
Chris Lu
89eb05b50f filer: support TTL for all filer stores 2020-03-09 01:02:01 -07:00
Chris Lu
2e3f6ad3a9 filer: remember content is gzipped or not 2020-03-08 21:39:33 -07:00
Chris Lu
1dc30214cb mark encryptVolumeData as not ready 2020-03-08 17:02:47 -07:00
Chris Lu
9b3109a5d8 filer: processing all response headers, no pass through to volume server
* filer calculate MD5 etag
* filer handle response headers, instread of pass it to volume servers
2020-03-08 15:42:44 -07:00
Chris Lu
936e7cdbfb pass in filer.copy cipher option 2020-03-07 12:46:00 -08:00
Chris Lu
1ae83c2938 go fmt 2020-03-07 06:12:57 -08:00
Chris Lu
8c3e25b38a mimee type default "application/octet-stream" to empty string 2020-03-07 06:08:08 -08:00
Chris Lu
13e215ee5c filer: option to encrypt data on volume server 2020-03-06 00:49:47 -08:00
Chris Lu
31c481e3fc fix typo 2020-03-05 16:00:20 -08:00
Chris Lu
e031605248 purge code 2020-03-04 00:44:27 -08:00
Chris Lu
f90c43635d refactoring 2020-03-04 00:39:47 -08:00
Chris Lu
6a8484b4ae master able to list all master clients by type 2020-03-01 22:13:47 -08:00
Chris Lu
1df13d0d6d adjust instructions 2020-02-27 13:15:21 -08:00
Chris Lu
555413d9fc weed queue starts 2020-02-27 00:07:13 -08:00
Chris Lu
556dd76303 s3: wait to connect to filer 2020-02-26 16:49:47 -08:00