Chris Lu
23014b6810
Merge branch 'master' into support_ssd_volume
2020-12-16 08:26:51 -08:00
Chris Lu
3c7f9633eb
show error a little bit more obviously
...
fix https://github.com/chrislusf/seaweedfs/issues/1678
2020-12-14 09:39:33 -08:00
Chris Lu
f6a419c26c
disk type configurable for each folder
2020-12-13 22:49:56 -08:00
Chris Lu
7c52a35942
refactor
2020-12-13 22:29:52 -08:00
Chris Lu
8baba93fce
rename parameter name to "disk"
2020-12-13 12:06:21 -08:00
Chris Lu
51eadaf2b6
rename parameter name to "disk"
2020-12-13 12:05:31 -08:00
Chris Lu
0d2ec832e2
rename from volumeType to diskType
2020-12-13 11:59:32 -08:00
Chris Lu
d156c74ec0
volume server set volume type and heartbeat to the master
2020-12-13 03:11:24 -08:00
Chris Lu
e9cd798bd3
adding volume type
2020-12-13 00:58:58 -08:00
Chris Lu
06bb7bf6c0
adding /usr/local/etc/seaweedfs for bsd style config search path
...
related to https://reviews.freebsd.org/D27391
2020-12-07 16:46:48 -08:00
Chris Lu
263eb29e9f
filer: add option to cache small files to filer store
2020-12-04 22:39:43 -08:00
Chris Lu
6d30b21b10
volume: add "-dir.idx" option for separate index storage
...
fix https://github.com/chrislusf/seaweedfs/issues/1265
2020-11-27 03:17:10 -08:00
Chris Lu
983fd90010
marked as WIP
2020-11-26 15:08:52 -08:00
Chris Lu
b11449f955
pass in dir.idx parameter to DiskLocation
2020-11-26 14:59:03 -08:00
Chris Lu
ef3e068c51
deprecating "filer.options.buckets_fsync" in filer.toml
2020-11-16 16:56:49 -08:00
Chris Lu
827930e249
Merge pull request #1610 from kmlebedev/prefer_read_in_this_dc
...
prefer to read from volumes in this data center
2020-11-11 22:09:41 -08:00
Konstantin Lebedev
0880aff224
update dc flag desc
2020-11-12 10:07:52 +05:00
Konstantin Lebedev
fc7baef5bb
fiil serverUrls sorted by data center
2020-11-12 02:13:33 +05:00
Chris Lu
745ee8d8f3
avoid unnecessary error message
...
fix https://github.com/chrislusf/seaweedfs/issues/1611
2020-11-11 13:00:05 -08:00
Chris Lu
8750cac090
move to util.RetryWaitTime
2020-11-01 02:36:43 -08:00
Chris Lu
de86945aeb
go fmt
2020-10-31 16:45:38 -07:00
Chris Lu
8826601be1
mount: optional limit for the number of concurrent writers
2020-10-30 21:22:20 -07:00
Chris Lu
7ecbb4b3c9
clean up
2020-10-30 10:46:31 -07:00
Kenny
9cc6575ab5
Update filer_copy.go
...
修改 weed filer.copy 上传目录时文件名变子目录的问题
2020-10-30 18:16:49 +08:00
Chris Lu
47047516d6
Revert "increase default volume file size limit to 1024"
...
This reverts commit 0983060a
I am confused why this was changed to 1024 by myself. 1GB should be too large for most cases.
2020-10-29 15:46:29 -07:00
Chris Lu
c6d9974fe7
server: Add option to enable/disable volume server
...
fix https://github.com/chrislusf/seaweedfs/issues/1572
2020-10-29 00:24:18 -07:00
Chris Lu
e219c57849
passing full path when assign volume locations
2020-10-25 15:46:29 -07:00
Chris Lu
f375b93aef
renaming
2020-10-25 15:32:43 -07:00
Chris Lu
63b0fb54f7
minor
2020-10-25 14:15:53 -07:00
Chris Lu
147d4d23b6
filer.copy: automatically use the right bucket
2020-10-25 11:21:33 -07:00
Chris Lu
da36abf033
go fmt
2020-10-24 20:12:04 -07:00
Chris Lu
b7b1eb51b1
default to 2 for better performance in common computer hardware
2020-10-24 20:08:26 -07:00
Chris Lu
1b17f71939
adjust election timeout to 10 seconds
2020-10-23 23:06:44 -07:00
Chris Lu
0cea84e7ab
make it easy with setting up multiple masters in a clcuster
2020-10-23 18:18:46 -07:00
Konstantin Lebedev
a9a7005687
set desc of option
2020-10-22 11:23:00 +05:00
Chris Lu
6c92fa349d
filer: support default rack
...
fix https://github.com/chrislusf/seaweedfs/issues/1546
2020-10-20 17:41:39 -07:00
Chris Lu
ee1fc6558a
refactor
2020-10-16 11:02:44 -07:00
Chris Lu
3f7d1d1bf1
Only wait on retryable requests
2020-10-13 00:29:46 -07:00
Chris Lu
7704469d29
mount: outsideContainerClusterMode changed to use volume server publicUrl
2020-10-11 20:42:15 -07:00
Chris Lu
d155f907c2
mount: configurable read wait time
2020-10-10 20:09:43 -07:00
Chris Lu
78859e35fc
s3: adjust config examples
2020-10-08 10:12:09 -07:00
Chris Lu
a8624c2e4f
read from alternative replica
...
related to https://github.com/chrislusf/seaweedfs/issues/1512
2020-10-07 22:49:04 -07:00
Chris Lu
da4edf3651
master: check peers for existing leader before starting a leader election
...
fix https://github.com/chrislusf/seaweedfs/issues/1509
2020-10-07 01:25:39 -07:00
Chris Lu
c543762e23
volume: auto expands -max setting
...
I have tried to run weed volume but got errors...
cmd:
weed volume -max=0 -mserver="127.0.0.1:9333" -port=8080 -dir="/var/www/d1/sfs,/var/www/d2/sfs,/var/www/d3/sfs"
log:
I1006 15:09:26 17040 file_util.go:23] Folder /var/www/d1/sfs Permission: -rwxr-xr-x
I1006 15:09:26 17040 file_util.go:23] Folder /var/www/d2/sfs Permission: -rwxr-xr-x
I1006 15:09:26 17040 file_util.go:23] Folder /var/www/d3/sfs Permission: -rwxr-xr-x
F1006 15:09:26 17040 volume.go:142] 3 directories by -dir, but only 1 max is set by -max
What am I doing wrong? Thx.
2020-10-06 09:05:30 -07:00
Chris Lu
2c20ef72ae
fix starting problem
2020-10-03 13:56:38 -07:00
Chris Lu
d77e3c96e9
FUSE: use the mount directory permissions
2020-10-03 13:37:35 -07:00
Устюжанин Антон Александрович
6fbfe73683
fix: resumeState
2020-10-03 22:16:47 +05:00
Устюжанин Антон Александрович
dc31b19469
fix: restore raft state
2020-10-03 14:03:41 +05:00
Устюжанин Антон Александрович
8c82fb7e5f
fix: restore raft state
2020-10-02 23:01:20 +05:00
Chris Lu
79ab10e300
adjust help message
2020-09-30 09:32:00 -07:00
Chris Lu
55cb68c09c
filer: replication follows master config if not specified
2020-09-30 09:15:55 -07:00
Chris Lu
e6552b5e1e
filer: able to start s3 together
2020-09-27 23:00:43 -07:00
Chris Lu
8516517c48
filer store: Cassandra supports username/password
2020-09-24 21:31:06 -07:00
Chris Lu
043b063136
Merge pull request #1482 from hilimd/master
...
Fix: s3 delete object
2020-09-24 18:21:34 -07:00
limd
48c578410f
mount: rollback default value
2020-09-25 09:18:52 +08:00
Chris Lu
4856bce0ee
adjust for metrics port
2020-09-24 10:21:23 -07:00
Chris Lu
d2d3aec3e1
consolidate to one metricsPort in "weed server" mode
2020-09-24 09:55:02 -07:00
Chris Lu
2e9099369e
Merge pull request #1489 from kmlebedev/promhttp
...
Promhttp
2020-09-24 09:50:48 -07:00
Konstantin Lebedev
98e9de6e11
fix style
2020-09-24 17:48:39 +05:00
Konstantin Lebedev
324e44d4b3
add start metrics server
2020-09-24 17:45:39 +05:00
limd
cbca14edc5
mount: fix k8s pvc and os mount directory permission bug
2020-09-24 18:07:16 +08:00
limd
59e91e9c7e
mount: fix k8s pvc mount directory permission
2020-09-24 10:25:45 +08:00
limd
9b21ec27e4
mount: auto created dir set corrct umask fix bug
2020-09-23 18:35:37 +08:00
Chris Lu
f7a0ccb595
mount: auto created dir set corrct umask
2020-09-23 02:31:19 -07:00
James Hartig
658fc2e5b6
Allow option to enable volume pprof on server
2020-09-21 22:43:10 -04:00
Chris Lu
d013d09a9b
adjust logging
2020-09-20 15:38:59 -07:00
Chris Lu
2c21eb1971
volume: get metrics configuration from master
...
fix https://github.com/chrislusf/seaweedfs/issues/1354
2020-09-19 00:03:00 -07:00
Chris Lu
6544e60bea
s3 add metrics
...
empty for now
2020-09-17 06:56:15 -07:00
Chris Lu
feca07bf96
Filer: ip bind
...
fix https://github.com/chrislusf/seaweedfs/issues/1470
2020-09-16 10:26:08 -07:00
Chris Lu
684a875876
update help message
2020-09-15 00:40:38 -07:00
Chris Lu
103fafe00b
weed export: print out [start,stop) content range
2020-09-14 22:57:23 -07:00
Chris Lu
5d6753fb98
shell: add volumeServer.leave command
2020-09-13 21:25:51 -07:00
Chris Lu
1af95c5b76
refactoring
2020-09-13 12:41:26 -07:00
Chris Lu
1a7afe7e6a
adjust default value
...
related to https://github.com/chrislusf/seaweedfs/issues/1453
2020-09-12 12:46:42 -07:00
Chris Lu
f2723c1bc8
do not idx file format
...
revert c9ab8d05fa
2020-09-12 12:42:36 -07:00
Chris Lu
446e476a11
go fmt
2020-09-12 04:08:03 -07:00
Chris Lu
cd9b89ba55
reduce default wait time to 10s before shutting down
2020-09-12 04:05:33 -07:00
Chris Lu
1b8094ef75
weed export: export deleted files
2020-09-11 02:05:14 -07:00
Chris Lu
c9ab8d05fa
fixes for reading deleted fid
2020-09-10 14:42:52 -07:00
Chris Lu
0d9b858cfb
Merge pull request #1455 from taozix/master
...
add more basic elastic options.
2020-09-10 01:31:17 -07:00
ruitao.liu
719dc43af1
modify elastic urls from string to array.
2020-09-10 16:24:09 +08:00
ruitao.liu
3f7fbfddca
add more basic elastic options.
2020-09-10 14:22:07 +08:00
Chris Lu
47b3f932e4
watch: adjust output format
2020-09-09 22:34:48 -07:00
Chris Lu
7f69acd1f2
sync pprof
2020-09-09 11:33:52 -07:00
Chris Lu
387ab6796f
filer: cross cluster synchronization
2020-09-09 11:21:23 -07:00
Chris Lu
4fc0bd1a81
return http response directly
2020-09-09 03:53:09 -07:00
ruitao.liu
83080b5e03
ES backended filer support kv ops.
2020-09-04 15:40:13 +08:00
Chris Lu
46f65a84a6
filer: elastic7 adjust default value
2020-09-03 20:12:38 -07:00
ruitao.liu
a93d27d1e8
new filer option to es v7.
2020-09-03 16:34:58 +08:00
Chris Lu
7e1aad0b54
mount: map uid/gid between local and filer
2020-09-03 00:07:22 -07:00
Chris Lu
63ad1abcce
watch: follow changes for a pattern
2020-08-28 19:43:04 -07:00
Chris Lu
6bc618051c
log message
2020-08-26 09:32:56 -07:00
Chris Lu
9dae4e9d3e
fix logs
2020-08-26 09:31:11 -07:00
Chris Lu
4c1a3187bf
fix log error
2020-08-26 09:25:54 -07:00
Konstantin Lebedev
464d4c82ec
stop send heartbeat before stop volume server
2020-08-26 16:16:11 +05:00
Chris Lu
fe01191b5b
support read option readDeleted=true
2020-08-18 19:22:16 -07:00
Chris Lu
6a92f0bc7a
refactoring to typed Size
...
Go is amazing with refactoring!
2020-08-18 17:04:28 -07:00
Chris Lu
0983060a90
increase default volume file size limit to 1024
...
avoid possible large chunk size set on mount or filer
2020-08-13 09:07:22 -07:00
Chris Lu
eed525b717
FUSE mount: remove DirListCacheLimit
...
outdated parameter
2020-07-28 09:24:39 -07:00
Chris Lu
a566bfc6e1
s3: use bucket in the domain
...
fix https://github.com/chrislusf/seaweedfs/issues/1405
2020-07-28 08:47:27 -07:00
Chris Lu
f43146b237
resolve directories if containing home directory
2020-07-16 22:50:14 -07:00
Chris Lu
befb396892
mount: resolve home directory
...
fix https://github.com/chrislusf/seaweedfs/issues/1391
2020-07-16 22:38:55 -07:00
Chris Lu
2b2ca7d1dc
stop if the directory can not be found
...
related to https://github.com/chrislusf/seaweedfs/issues/1391
2020-07-16 13:24:53 -07:00
Chris Lu
ddec7b2bb9
go fmt
2020-07-14 07:34:16 -07:00
李远军
6608cb5f43
Revert "remove fixJpgOrientation"
...
This reverts commit de5ca9b2
2020-07-10 10:08:36 +08:00
Chris Lu
a2eb680f34
filer: default filer store directory
...
1. set default filer store directory
2. set peers, avoiding empty string counted as 1.
2020-07-07 23:06:48 -07:00
Chris Lu
d198e8c6d6
filer: support cross filer meta data sync if sharing the same store
2020-07-06 09:24:54 -07:00
Chris Lu
59152334c7
default to 1%
2020-07-03 16:42:53 -07:00
Chris Lu
acf4a9ea99
volume: min free space default to 1%
2020-07-03 16:42:02 -07:00
Chris Lu
4d1484628a
refactoring
2020-07-03 16:41:30 -07:00
Chris Lu
12f6b683fc
default to 8 volumes
...
in case all the volumes are allocated for a collection
2020-06-28 14:34:21 -07:00
Chris Lu
9033a7d369
removed async option
2020-06-28 10:18:32 -07:00
Chris Lu
1bb8cae65d
reverting and working
2020-06-28 10:14:17 -07:00
Chris Lu
98493e0152
volume: add default minFreeSpacePercent
...
fix https://github.com/chrislusf/seaweedfs/issues/1366
2020-06-21 08:44:06 -07:00
Chris Lu
e7c36f740a
refactoring
2020-06-21 08:38:00 -07:00
Chris Lu
8b4a32e782
adjust help message
2020-06-21 08:34:24 -07:00
Chris Lu
b4ec04d464
fix the help message
2020-06-21 08:31:20 -07:00
Chris Lu
de5ca9b258
remove fixJpgOrientation
2020-06-20 12:50:40 -07:00
Chris Lu
ca3516ac6d
adjust protoc
2020-06-20 08:00:25 -07:00
Chris Lu
e912fd15e3
renaming
2020-06-19 22:45:27 -07:00
Chris Lu
b78a2dc59b
adjust help message
2020-06-19 12:55:09 -07:00
Chris Lu
48880edc2c
volume: fix typo in minFreeSpacePercent option
2020-06-14 20:37:05 -07:00
Chris Lu
a0464fc267
fix help
...
fix https://github.com/chrislusf/seaweedfs/issues/1359
2020-06-12 22:06:21 -07:00
Chris Lu
0d1d83a5a5
mount: add option to auto create dir to mount
2020-06-12 11:46:58 -07:00
Chris Lu
628b27ef3b
purge old cache implementation
2020-06-11 01:50:00 -07:00
Chris Lu
b9365de47b
fix volume server pprof
2020-06-10 13:10:10 -07:00
Kimbsen
03e8f976d9
added --pprof to volume servers
2020-06-10 11:52:24 +02:00
Chris Lu
45a9d852d3
go fmt
2020-06-05 15:27:10 -07:00
Evgenii Kozlov
2410ffbd79
Name fix
2020-06-05 19:00:38 +03:00
Evgenii Kozlov
0e0db70f55
Set volumes ReadOnly if low free disk space
2020-06-05 18:18:15 +03:00
Chris Lu
393ab07c7c
disable pulse seconds
...
there are a few other hard coded timing settings, which will cause disconnected volume servers.
2020-06-04 10:52:01 -07:00
Chris Lu
bc2ec6774d
inject git version into build
2020-06-02 00:10:38 -07:00
Chris Lu
ac48c899e8
add printouts for synchronizing metadata which can be slow for large folders
2020-05-18 10:31:12 -07:00
Chris Lu
0c856e8387
avoid possible wrong arguments
...
fix https://github.com/chrislusf/seaweedfs/issues/1320
2020-05-18 10:07:12 -07:00
Chris Lu
2bfd810912
combine msgBroker into weed server command
2020-05-17 22:44:20 -07:00
Chris Lu
39e72fb23c
go fmt
2020-05-10 03:50:30 -07:00
Chris Lu
1e3e4b3072
add broker connects to filer
2020-05-05 02:05:28 -07:00
Chris Lu
78422c2f69
Merge pull request #1298 from song-zhang/master
...
add fsync option for benchmark
2020-04-30 20:16:42 -07:00
Chris Lu
871efa4fc1
refactoring
...
some previous chunk etag was using md5, which should be wrong.
2020-04-30 17:20:47 -07:00
zhangsong
d819865fe7
add fsync option for benchmark
2020-04-30 23:31:08 +08:00
Chris Lu
eab4c9219c
randomize benchmark content
...
fix https://github.com/chrislusf/seaweedfs/issues/1294
2020-04-28 01:50:56 -07:00
Chris Lu
5c57297bd1
metadata log: read from any timestamp
2020-04-28 00:05:57 -07:00
wuyuxiang
6850d28d6b
refacotr: move signal handling and pprof to grace package
2020-04-28 14:10:23 +08:00
Chris Lu
0c2248f83a
add lock/unlock to example admin scripts
2020-04-23 18:41:48 -07:00
Chris Lu
913aa0f14e
mount: default to enable asyncMetaDataCaching
2020-04-23 14:14:28 -07:00
Chris Lu
b8e4238ad2
add placeholder for cached meta data reading
2020-04-21 18:02:08 -07:00
Chris Lu
57df14f76f
report error if copying failed
2020-04-21 14:28:46 -07:00
Chris Lu
cf54d70311
filer: add -ip.bind option
2020-04-21 14:21:06 -07:00
Chris Lu
f373232227
Merge pull request #1280 from bukton123/filer_mongodb
...
Add new filer support mongodb
2020-04-19 20:35:19 -07:00
bukton
3defa718a1
add default config filer mongodb to scaffold.go
2020-04-19 17:52:38 +07:00
Chris Lu
71ffb98475
broker: add profiling
2020-04-19 03:03:40 -07:00
Chris Lu
767f14dfcd
master, volume default to detected host
2020-04-18 15:20:20 -07:00
Chris Lu
076c8bd3bc
filer master start up with default ip address instead of just localhost
2020-04-18 15:17:27 -07:00
Chris Lu
11f5a6d913
messaging can compile now
2020-04-18 01:12:01 -07:00
Chris Lu
826f96b6f3
fix unreachable code
2020-04-17 10:04:28 -07:00
Chris Lu
2a45897237
broker: read cipher value from filer
2020-04-17 02:29:00 -07:00
Chris Lu
f5a748d33c
refactoring
2020-04-16 02:55:09 -07:00
Chris Lu
ce4b369be2
scaffolding messaging
2020-04-16 02:21:23 -07:00
Chris Lu
d8f5985e5e
add logs
2020-04-13 12:58:45 -07:00
Chris Lu
7764e0465c
refactoring
2020-04-12 21:00:55 -07:00
Chris Lu
6f948e4887
remove configurable topics folder location
2020-04-12 13:07:59 -07:00
Chris Lu
b5a713be68
filer store: add redis2 redis_cluster2 as default redis store
2020-04-12 02:50:41 -07:00
Chris Lu
211d87cf4c
mount: option to disable caching
2020-04-12 00:52:54 -07:00
Chris Lu
1c65656fb4
s3: add option to fsync buckets
2020-04-11 23:37:10 -07:00
Chris Lu
5987810e5e
set default chunk size to 16
2020-04-11 21:22:52 -07:00
Chris Lu
df97da25f9
mount: add on disk caching
2020-04-11 21:12:41 -07:00
Chris Lu
dce2702e1b
fix the instruction
2020-04-10 14:50:10 -07:00
Chris Lu
8db2120bee
add logs
2020-04-09 23:42:59 -07:00
Chris Lu
59f40e2027
volume: best effort to detect ip address
...
fix https://github.com/chrislusf/seaweedfs/issues/1264
2020-04-09 00:26:24 -07:00
Chris Lu
f6a7e79dc3
weed shell: simplify CLI option for filer
2020-04-08 23:57:15 -07:00
Chris Lu
2329d9e0c1
add volume.fix.replication to default scaffold
2020-04-08 23:36:22 -07:00
Chris Lu
4c498e73de
filer replication: add s3 endpoint
2020-04-07 17:49:00 -07:00
Chris Lu
2d43f85577
watch entries with common path prefix
2020-04-05 12:51:21 -07:00
Chris Lu
af1f64d244
change from 'weed tail' to 'weed watch'
2020-04-05 12:39:20 -07:00
Chris Lu
bf270d9e8c
filer: able to tail meta data changes
2020-04-05 00:51:16 -07:00
Chris Lu
d2f8e50459
make error a little bit more clearer
...
fix https://github.com/chrislusf/seaweedfs/issues/1259
2020-04-03 00:24:09 -07:00
Chris Lu
5f881d2fd5
mount: fail early rather than after mounted the directory
...
fix https://github.com/chrislusf/seaweedfs/issues/1258
2020-04-03 00:21:03 -07:00
James Hartig
eae3f27c80
Added treat_replication_as_minimums master toml option
2020-04-01 19:08:48 -04:00
Chris Lu
a75d50bbb8
FUSE: add configurable in memory chunk cache size
2020-03-28 14:07:16 -07:00
Chris Lu
eedd33dda3
refactoring
2020-03-28 13:41:58 -07:00
Chris Lu
40601953bf
comments
2020-03-23 02:54:49 -07:00
Chris Lu
daeb18be30
scaffold: fix master.toml typo
...
fix https://github.com/chrislusf/seaweedfs/issues/1247
2020-03-23 01:45:43 -07:00
Chris Lu
e32999108a
add auto configured volume max count help messge
2020-03-22 16:27:15 -07:00
Chris Lu
7c111f7b75
FUSE mount: make "nonempty" optional
...
https://github.com/chrislusf/seaweedfs/issues/1094
2020-03-21 19:14:25 -07:00
Chris Lu
709f231e23
tiered storage: add s3 endpoint for private s3 implementation
...
fix https://github.com/chrislusf/seaweedfs/issues/1238
2020-03-19 21:13:56 -07:00
Chris Lu
d848d08944
use public url also for deletions
2020-03-18 11:16:45 -07:00
Chris Lu
6566c8e114
weed upload: add usePublicUrl option
...
fix https://github.com/chrislusf/seaweedfs/issues/1236
2020-03-18 10:50:53 -07:00
Chris Lu
e85da50ad4
remove tikv, move its support to "tikv" branch
2020-03-11 20:59:00 -07:00
Chris Lu
d439d83772
volume: follow compactionBytePerSecond
...
related to https://github.com/chrislusf/seaweedfs/issues/1108
2020-03-11 10:32:17 -07:00
Chris Lu
e73b0c2457
remove not ready status
2020-03-09 22:31:14 -07:00
Chris Lu
0871d2cff0
volume: fix memory leak during compaction
...
fix https://github.com/chrislusf/seaweedfs/issues/1222
2020-03-09 22:29:02 -07:00
Chris Lu
89eb05b50f
filer: support TTL for all filer stores
2020-03-09 01:02:01 -07:00
Chris Lu
2e3f6ad3a9
filer: remember content is gzipped or not
2020-03-08 21:39:33 -07:00
Chris Lu
1dc30214cb
mark encryptVolumeData as not ready
2020-03-08 17:02:47 -07:00
Chris Lu
9b3109a5d8
filer: processing all response headers, no pass through to volume server
...
* filer calculate MD5 etag
* filer handle response headers, instread of pass it to volume servers
2020-03-08 15:42:44 -07:00
Chris Lu
936e7cdbfb
pass in filer.copy cipher option
2020-03-07 12:46:00 -08:00
Chris Lu
1ae83c2938
go fmt
2020-03-07 06:12:57 -08:00
Chris Lu
8c3e25b38a
mimee type default "application/octet-stream" to empty string
2020-03-07 06:08:08 -08:00
Chris Lu
13e215ee5c
filer: option to encrypt data on volume server
2020-03-06 00:49:47 -08:00
Chris Lu
31c481e3fc
fix typo
2020-03-05 16:00:20 -08:00
Chris Lu
e031605248
purge code
2020-03-04 00:44:27 -08:00
Chris Lu
f90c43635d
refactoring
2020-03-04 00:39:47 -08:00
Chris Lu
6a8484b4ae
master able to list all master clients by type
2020-03-01 22:13:47 -08:00
Chris Lu
1df13d0d6d
adjust instructions
2020-02-27 13:15:21 -08:00
Chris Lu
555413d9fc
weed queue starts
2020-02-27 00:07:13 -08:00
Chris Lu
556dd76303
s3: wait to connect to filer
2020-02-26 16:49:47 -08:00
Chris Lu
0156e2975a
mount: add mode to run external to SeaweedFS container cluster
2020-02-26 16:46:01 -08:00
Chris Lu
543cf1c80e
fix compilation error
2020-02-25 23:59:30 -08:00
Chris Lu
b1f377f822
mount: fail fast when connecting to filer
...
fix https://github.com/chrislusf/seaweedfs/issues/1034
2020-02-25 23:51:34 -08:00
Chris Lu
97ab8a1976
remove ctx if possible
2020-02-25 22:23:59 -08:00
Chris Lu
892e726eb9
avoid reusing context object
...
fix https://github.com/chrislusf/seaweedfs/issues/1182
2020-02-25 21:50:12 -08:00
Chris Lu
0841bedb15
move filer assign volume grpc errror to response
2020-02-25 17:15:09 -08:00
Chris Lu
6ab7368ef2
filer: dynamically create bucket under /buckets folder
2020-02-24 22:28:45 -08:00
Chris Lu
5bcb44eda9
filer: move buckets folder to filer.toml since it should not be changed often
2020-02-24 14:42:57 -08:00
Chris Lu
d8dec2323b
s3: move buckets folder configuration to filer
2020-02-24 14:34:14 -08:00
Chris Lu
4ed6b584e2
s3: access control limited by bucket
2020-02-22 21:34:18 -08:00
Chris Lu
b07089cff2
webdav out of "unstable"
2020-02-18 19:38:59 -08:00
Chris Lu
0a57db7026
Merge pull request #1200 from chrislusf/add_s3_acl
...
Add s3 ACL
2020-02-17 12:53:30 -08:00
Chris Lu
cf5064d702
properly close http response
2020-02-14 09:09:15 -08:00
Chris Lu
9b6296e77a
volume: add grpc file read operation
...
This is added more for performance benchmarking.
2020-02-14 00:37:32 -08:00
Chris Lu
b90ad6f452
add v2 support
2020-02-09 16:02:05 -08:00
Chris Lu
e7b63d50b1
add s3.config to server.go
2020-02-09 14:31:51 -08:00
Chris Lu
9ed364f053
support acl
2020-02-09 14:30:02 -08:00
Chris Lu
dc786a63ac
master: add configurable volume growth toml setting
2020-02-03 18:15:16 -08:00
Chris Lu
8925f3305d
adjust example
2020-01-29 09:11:07 -08:00
Chris Lu
d335f04de6
support env variables to overwrite toml file
2020-01-29 09:09:55 -08:00
Chris Lu
72a64a5cf8
use the same context object in order to retry
2020-01-26 14:42:11 -08:00
Chris Lu
c48fc8b4de
grpc send error via response instead of grpc error
2020-01-25 09:17:19 -08:00
Chris Lu
e4b660699f
correct comments
...
fix https://github.com/chrislusf/seaweedfs/issues/1181
2020-01-12 20:31:33 -08:00
Chris Lu
dc9457fb87
fix compilation error
2020-01-07 09:03:47 -08:00
Chris Lu
aa73364a24
weed fix: add back .idx file generation
2020-01-06 16:29:59 -08:00
Chris Lu
3eafec4b29
volume: add option to limit file size
2020-01-03 00:37:26 -08:00
Chris Lu
86d4b18eb5
filer: optional recursive deletion
...
fix https://github.com/chrislusf/seaweedfs/issues/1176
2019-12-31 11:52:54 -08:00
Chris Lu
efd2f50ede
compaction changed to .idx based deletion
2019-12-24 14:55:50 -08:00
Chris Lu
abffe857a1
change btree map to in memory level db
2019-12-24 10:18:56 -08:00
Chris Lu
09ca936c78
shell: add ec.decode command
2019-12-23 12:48:20 -08:00
Chris Lu
a9b1ee18af
default redis reads options
...
details in https://github.com/chrislusf/seaweedfs/pull/1165
2019-12-21 09:30:51 -08:00
Chris Lu
fcbd04c425
default to false in scaffold.go
2019-12-21 08:31:27 -08:00
Dakota Sullivan
f934a85d58
Add readOnly and routeByLatency options to scaffold.go
2019-12-20 19:27:15 -07:00
Chris Lu
392678f8f3
upload skipping mimetype if not needed
2019-12-15 20:57:08 -08:00
Chris Lu
f59119cd1d
mount: rename to dirListCacheLimit to configure cache for long format directory list
2019-12-13 10:22:24 -08:00
Chris Lu
05c3b795dc
filer: configurable directory list cache size
2019-12-13 10:05:43 -08:00
Chris Lu
bbb6ebc3c0
filer: DeleteFolderChildren for deleting large folders
2019-12-13 00:23:05 -08:00
Chris Lu
5847c5afff
return meaningful error log
2019-12-10 22:09:29 -08:00
Chris Lu
0da7b894cc
pass backend config from master to volume servers
2019-11-29 01:05:09 -08:00
Chris Lu
a9dbd52e06
Merge pull request #1145 from stlpmo-jn/volume_graceful_stop
...
let volume server graceful stop
2019-11-29 00:11:57 -08:00
stlpmo
ea9d1ebd2f
resolve the Duration.Milliseconds() is not exist in golang v11 & v12
2019-11-29 15:17:17 +08:00
Chris Lu
61bc1d6ffd
tweaking
2019-11-28 18:47:51 -08:00
Chris Lu
09c6d8fa32
fix test
2019-11-28 18:44:27 -08:00
Chris Lu
f60154f330
master load backend storage config from master.toml
2019-11-28 18:33:18 -08:00
Chris Lu
641b92f53c
fix scaffold error
2019-11-28 18:04:25 -08:00
Chris Lu
0b822a6fb5
Merge pull request #1142 from iliul/wip-fuse-mount
...
fuse: check mount point available before do runmount
2019-11-27 19:22:51 -08:00
Lei Liu
13b6f09fea
fuse: adjust linux specific logic
...
Signed-off-by: Lei Liu <lei01.liu@horizon.ai>
2019-11-28 10:21:31 +08:00
Chris Lu
46294b42e6
filer: support memsql
...
fix https://github.com/chrislusf/seaweedfs/issues/1141
2019-11-27 12:34:03 -08:00
Lei Liu
1ba3456ac2
go fmt
...
Signed-off-by: Lei Liu <liul.stone@gmail.com>
2019-11-27 23:08:51 +08:00
Lei Liu
4c87b222f1
fuse: check mount point available before do runmount
...
1. Use more readable mount point information
2. Fix some typos
eg:
$ df -Th
Filesystem Type Size Used Avail Use% Mounted on
localhost:8888:/ fuse.seaweedfs 206G 512 206G 1% /mnt/weedfs
$ mount | grep weedfs
localhost:8888:/ on /mnt/weedfs type fuse.seaweedfs (rw,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other)
Signed-off-by: Lei Liu <lei01.liu@horizon.ai>
2019-11-27 22:56:44 +08:00
Chris Lu
67e5f5b558
typo
2019-11-27 03:09:45 -08:00
Chris Lu
9711a6ffaa
WIP
2019-11-18 19:24:37 -08:00
stlpmo
753138a3f7
test passed
2019-11-17 11:40:36 +08:00
Chris Lu
de0c983808
refactoring scaffolding for master.toml
2019-11-14 22:20:38 -08:00
Lei Liu
46755ea1e1
fix master maintenance logic
...
Signed-off-by: Lei Liu <lei01.liu@horizon.ai>
2019-11-12 14:46:10 +08:00
Chris Lu
ee90236a97
Merge branch 'master' into refactoring_dat_backend
2019-11-11 21:30:14 -08:00
stlpmo
62d393d6c9
ut pass
2019-11-11 10:52:21 +08:00
stlpmo
802a0eb3fe
move from cmd-line to scaffold
2019-11-11 09:15:17 +08:00
stlpmo
1c8bed3810
delete the var etcdClient and comments
...
refactor the code
add sequencer cmd-line
delete nerver used codes
2019-11-05 15:45:25 +08:00
Chris Lu
5b950c735e
Merge branch 'master' into refactoring_dat_backend
2019-10-29 23:18:41 -07:00
Lei Liu
c2884cace2
misc updated
...
Signed-off-by: Lei Liu <lei01.liu@horizon.ai>
2019-10-29 21:28:28 +08:00
Chris Lu
19b6a16003
changed from os.file to backend.DataStorageBackend
2019-10-29 00:35:16 -07:00
Chris Lu
eb2172f63f
FUSE: add os specific mount options
...
related to https://github.com/chrislusf/seaweedfs/issues/1094
2019-10-27 09:12:10 -07:00
Chris Lu
69de05f6cb
go fmt
2019-10-25 07:45:12 -07:00
Chris Lu
288c45a690
fix iitial filer url
2019-10-25 07:44:37 -07:00
Chris Lu
05fe7a2366
Merge pull request #1092 from binbinshi/master
...
fix: weed shell filer config is constant
2019-10-25 07:08:59 -07:00
shibinbin
fe1f469978
fix : change default filer.port to 8888
2019-10-25 17:39:29 +08:00
shibinbin
3464f21b5c
fix: weed shell add filer runtime attributes
2019-10-25 17:34:03 +08:00
Chris Lu
9573c0f4b3
filer: filer.copy cleanup in case of failed uploads
2019-10-24 10:16:01 -07:00
Chris Lu
5d78871487
filer: parallize weed filer.copy for single file
2019-10-24 07:26:23 -07:00
Chris Lu
23240efa63
filer: add tikv support
2019-10-23 00:31:31 -07:00
Chris Lu
fc412e428b
refactor ScanVolumeFileFrom()
2019-10-22 00:50:30 -07:00
Chris Lu
c262526d8a
Merge pull request #1087 from joeslay/master
...
Storing files in in-memory collections on windows
2019-10-21 22:30:35 -07:00
zhangsong
65f1095a54
fix benchmark cmd usage info
2019-10-14 20:23:32 +08:00
j.laycock
6fc6322c90
Change joeslay paths to chrislusf paths
2019-09-12 14:18:21 +01:00
Tom Maxwell
4a878c0006
Changed the InMemory bool to a uint32 so that it can be used to alter how much space to reserve
2019-09-04 15:27:14 +01:00
j.laycock
1f01eb78e8
Rename mem_map to mMap, remove some in_memory variables being passed around, added MemoryMapped member to volume struct
2019-09-03 17:00:59 +01:00
Tom Maxwell
d637d86d22
Changes to try and pass the URL parameters through - in memory flag not working still
2019-09-03 15:41:28 +01:00
j.laycock
595a1beff0
Swap imports to use joeslay
2019-09-02 11:28:40 +01:00
xushuxun
5904d78bd4
weed backup: add ttl and replication parameter
2019-08-16 11:05:22 +08:00
Chris Lu
6f75df8660
server: fix starting default master list
2019-08-10 03:10:03 -07:00
Chris Lu
b9f6236f8a
filer: fix typo
...
fix https://github.com/chrislusf/seaweedfs/issues/1031
2019-08-06 00:42:19 -07:00
Chris Lu
c79274f7eb
notes on supported meta data store
2019-08-05 16:45:21 -07:00
Feng Ye
fc0498798b
Add etcd filestore
2019-08-01 10:16:45 +08:00
Chris Lu
8afd8d35b3
master: followers can also lookup and redirect
...
improve scalability
2019-07-28 03:58:13 -07:00
Chris Lu
2c6cf72e73
refactoring
2019-07-28 01:55:05 -07:00
Chris Lu
344ce90315
remove weedfuse
2019-07-24 00:42:12 -07:00
Chris Lu
5956dfd08d
mount: add umask option
...
related to https://github.com/chrislusf/seaweedfs/issues/978
2019-07-24 00:03:05 -07:00
Chris Lu
61b7a650f4
filer.copy default to 32MB chunks
2019-07-10 21:48:27 -07:00
Chris Lu
23d36dfbaf
1.40
...
enable azure replication
2019-07-02 21:27:05 -07:00
Chris Lu
88fed85bba
temp skipping azure
...
can not release due to https://github.com/Azure/azure-pipeline-go/issues/23
2019-06-30 23:17:45 -07:00
Chris Lu
15ecf4e3bf
filer: default to leveldb2
2019-06-30 00:44:57 -07:00
Chris Lu
c777102da9
go fmt
2019-06-23 15:30:16 -07:00
Chris Lu
d5560f2705
simplify metrics settings
2019-06-23 15:29:49 -07:00
Chris Lu
6f8b335007
fix duplicated flags
2019-06-23 03:11:21 -07:00
Chris Lu
f16375621f
big refactoring
2019-06-23 03:08:27 -07:00
Chris Lu
8da5d5b094
filer.copy: use filer settings, avoid unnecessary command line options
...
fix https://github.com/chrislusf/seaweedfs/issues/968
2019-06-23 01:57:35 -07:00
Chris Lu
d8ed73926d
volume servers get metrics address and interval from the master
2019-06-17 14:51:47 -07:00
Chris Lu
8b43679ae3
refactoring
2019-06-15 12:21:44 -07:00
Chris Lu
5f6c9825f8
volume server: adds basic metrics
2019-06-14 00:54:56 -07:00
Chris Lu
a11525fe4e
filer: adds basic metrics pushing to Prometheus gateway
2019-06-13 02:01:54 -07:00
Chris Lu
e66bddd84c
customizable sleep intervals
2019-06-06 00:39:08 -07:00
Chris Lu
50aa769554
jwt for read access control
2019-06-06 00:29:02 -07:00
Chris Lu
784141c5e6
adjust script parameter
2019-06-05 01:48:03 -07:00
Chris Lu
ede876cfdb
periodic scripts exeuction from leader master
2019-06-05 01:30:24 -07:00
Chris Lu
3f9ecee40f
working with reading remote intervals
2019-05-28 21:29:07 -07:00
Chris Lu
17ac1290c0
volume: load ec shards during heartbeats to master
2019-05-21 22:41:20 -07:00
Chris Lu
54b835e1ae
filer: redis cluster add option for password
...
fix https://github.com/chrislusf/seaweedfs/issues/971
2019-05-20 09:00:30 -07:00
Chris Lu
0094ca3f8a
mark the webdav as unstable
2019-05-19 21:08:26 -07:00
Chris Lu
0fe286a6ca
filer: optimization for leveldb, add leveldb2
...
large filer db may see high CPU and disk usage due to background compaction
2019-05-17 17:33:49 -07:00
Chris Lu
7e0c3bb732
filer: adjust recommended mysql meta data type to LONGBLOB
...
fix https://github.com/chrislusf/seaweedfs/issues/966
2019-05-17 09:11:08 -07:00
Chris Lu
ab22407763
filer ip only use bind ip
2019-05-14 23:43:40 -07:00
Chris Lu
4dfcd2169e
weedfuse: adapt to fstab command line pattern
2019-05-11 01:17:55 -07:00
Chris Lu
9a4fb14ea0
weedfuse: daemonize
2019-05-10 17:08:15 -07:00
Chris Lu
1ca1ec906a
add weedfuse
2019-05-10 15:03:31 -07:00
Chris Lu
d01807ba05
shell: volume.balance add option for all_collections, each_collection
2019-05-06 14:12:19 -07:00
Chris Lu
4c392ca762
use ip.bind to listen for grpc calls
...
help to avoid https://github.com/chrislusf/seaweedfs/issues/952
2019-05-06 08:10:18 -07:00
Chris Lu
25941e0500
master: add jwt expires_after_seconds
2019-05-04 08:42:25 -07:00
Chris Lu
3fbe22b203
better error handling for nil raftServer
...
addressing https://github.com/chrislusf/seaweedfs/issues/717
2019-05-03 22:33:08 -07:00
Chris Lu
b335f81a4f
volume: add option to limit compaction speed
2019-05-03 17:22:39 -07:00
Chris Lu
9d3c945b00
Merge pull request #950 from chrislusf/webdav
...
add WebDav
2019-05-03 01:25:10 -07:00
Chris Lu
b30c14b631
webdav: can read now
2019-05-03 00:24:35 -07:00
Chris Lu
47f14775d7
scaffolding, but seems hard to satisfy mac, windows, and different os versions
...
https://www.reddit.com/r/golang/comments/3wrxo3/webdav_server_that_provide_filesystem_using/
2019-05-02 14:22:10 -07:00
Wine93
32f93fb09e
style: go fmt for all
2019-04-30 03:22:19 +00:00
Chris Lu
82e06a780b
mount: allow nonempty fuse mount to work with Samba
...
possibly fix https://github.com/chrislusf/seaweedfs/issues/936
2019-04-29 10:42:09 -07:00
Chris Lu
9774ede3cf
adjust usage description
2019-04-24 00:25:20 -07:00
Chris Lu
b04d7e3ac0
weed server: also optionally start S3 gateway
2019-04-24 00:18:01 -07:00
Chris Lu
dabc9c9a89
filer: listen on specified ip address
2019-04-24 00:01:39 -07:00
Chris Lu
21fe5bde7c
filer: remove customizable grpc port, fixed to filer port + 10000
2019-04-23 23:58:52 -07:00
Chris Lu
3b3651dea3
volume: atomic copying file, adds version and stopOffset
2019-04-19 12:29:49 -07:00
Chris Lu
730a032137
filer mysql: adjust scaffolding instructions
2019-04-19 09:55:46 -07:00
Chris Lu
ac2727853f
fix needle map entry size
2019-04-19 00:39:34 -07:00
Chris Lu
e5506152c0
refactoring
2019-04-18 21:43:36 -07:00
Chris Lu
a2d34d4802
filer.copy: add adjustable compression level
2019-04-18 20:21:28 -07:00
Chris Lu
13ad5c1966
refactoring
2019-04-17 22:04:49 -07:00
Chris Lu
8ea1ee6dfa
weed shell: add fs.meta.notify, removing filer.export
2019-04-16 01:58:28 -07:00
Chris Lu
2e03f3adef
filer: rename -port.public to -port.readonly
2019-04-11 20:42:55 -07:00
Wine93
361912224d
typo: remove blank
2019-04-11 09:18:53 +00:00
Chris Lu
c0d6cc0067
weed mount: mount to the same folder name
2019-04-10 23:25:41 -07:00
Chris Lu
3555628ad5
weed volume: fix bug, remove boltdb, btree
2019-04-09 10:08:59 -07:00
Chris Lu
8a5ce16e96
weed volume: remove boltdb, btree options, add options for leveldb medium, large
2019-04-09 09:42:06 -07:00
Chris Lu
88ba08cb9e
add large disk support
2019-04-08 22:32:42 -07:00
Chris Lu
000ee725fc
refactor Offset into a struct of bytes
2019-04-08 19:40:56 -07:00
Chris Lu
a41ba79119
Merge pull request #914 from jba/gocdk-receive
...
replication: add GoCDK PubSub support
2019-04-08 10:31:02 -07:00
Chris Lu
d14b614407
weed filer.copy: use existing file owner and gropu id
2019-04-07 11:31:50 -07:00
Chris Lu
c731d9be6d
weed filer.copy: add option to profile the execution
2019-04-07 09:13:24 -07:00
Chris Lu
24c020104a
weed filer.copy: parallelize the copying to increase throughput
...
improvement https://github.com/chrislusf/seaweedfs/issues/915
2019-04-05 23:35:30 -07:00
Jonathan Amsterdam
72920efc20
added entry to scaffold
2019-04-05 13:43:38 -04:00
Chris Lu
300b39b5af
weed filer.copy: use cached filer grpc connection
...
another attemp for https://github.com/chrislusf/seaweedfs/issues/915
2019-04-05 01:09:06 -07:00
Chris Lu
921231268b
weed filer.copy: skip files that can not be opened
...
such as socket
2019-04-05 00:04:00 -07:00
Chris Lu
4f714cef22
default weed filer.copy and weed upload chunk size to 32MB
2019-04-04 23:14:32 -07:00
Chris Lu
af37b374cb
weed shell: fs.cd change current directory
2019-04-04 19:27:51 -07:00
Chris Lu
715a38da1e
weed shell: add fs.cd, fs.pwd to change to a directory and print current directory
2019-04-03 00:20:00 -07:00
Chris Lu
2a52e70a41
weed filer: increase default mysql/postgres dir or name length to 65535
2019-04-01 12:37:54 -07:00
Chris Lu
920b4e56aa
refactoring
2019-03-29 22:53:35 -07:00
Chris Lu
f60d383947
weed filer.copy: support multiple masters
2019-03-29 22:50:14 -07:00
Chris Lu
487ca51491
weed filer.export: fix pagination error
2019-03-28 00:46:15 -07:00
Chris Lu
7a14cdc90c
refactoring, go fmt
2019-03-25 23:18:40 -07:00
Chris Lu
df95ce0b6c
weed backup: efficient delta backup
...
fix https://github.com/chrislusf/seaweedfs/issues/399
2019-03-25 23:01:53 -07:00
Chris Lu
70815e9124
WIP
2019-03-25 09:16:12 -07:00
Chris Lu
a3490b600c
weed filer, weed master: add option to disable http
2019-03-21 16:00:46 -07:00
Jonathan Amsterdam
e966033098
Merge branch 'master' of https://github.com/chrislusf/seaweedfs into gocdk
2019-03-20 16:02:38 -04:00
Jonathan Amsterdam
8db82e2b75
notification: add Go CDK pubsub support
...
Add the gocdk_pub_sub package, which supports the Go Cloud Development
Kit pubsub API.
Link in all current providers.
Update the notification scaffold.
2019-03-20 07:57:58 -04:00
chenwanli
45a52b17fd
benchmark: add replication flag
2019-03-20 16:38:11 +08:00
Chris Lu
da871896c3
weed filer: set grpc port to port + 10000
2019-03-19 05:47:41 -07:00
Chris Lu
657dd2e6c9
add shell command to list all collections
2019-03-16 13:43:16 -07:00
Chris Lu
55bab1b456
add context.Context
2019-03-15 17:20:24 -07:00
Chris Lu
cece860bfd
add context to all filer APIs
2019-03-15 15:55:34 -07:00
Chris Lu
d30600e6d6
fix filer starting under "weed server" when ip is not localhost
...
fix https://github.com/chrislusf/seaweedfs/issues/886
2019-03-15 09:57:33 -07:00
Chris Lu
a4e8aa98a0
weed server: mkdir first, and set the master ip address
...
fix https://github.com/chrislusf/seaweedfs/issues/885
2019-03-15 00:32:42 -07:00
Chris Lu
6fe071175d
mount: add option to disable allow others
...
fix https://github.com/chrislusf/seaweedfs/issues/877
2019-03-10 19:45:40 -07:00
Chris Lu
4773497d2c
add back "/cluster/status"
...
fix https://github.com/chrislusf/seaweedfs/issues/870
2019-03-06 12:10:45 -08:00
Chris Lu
ab32556a51
weed server: fix error coming from moving raft to grpc
...
fix https://github.com/chrislusf/seaweedfs/issues/872
2019-03-06 10:21:49 -08:00
Chris Lu
2812c14520
master: add option to disable http operations
2019-03-02 05:10:05 -08:00
Chris Lu
7bbe24dd28
volume server directly support https
2019-02-25 00:43:36 -08:00
Chris Lu
0fa1be8c4b
adjust formatting and comments
2019-02-19 21:10:10 -08:00
Chris Lu
07af52cb6f
raft change from http to grpc
...
master grpc port is fixed to http port + 10000
2019-02-18 22:38:14 -08:00
Chris Lu
77b9af531d
adding grpc mutual tls
2019-02-18 12:11:52 -08:00
Chris Lu
98a03b38e5
avoid util package depends on security package
2019-02-16 12:23:35 -08:00
Chris Lu
74fb237727
benchmark can work in secure mode
2019-02-15 00:09:19 -08:00
Chris Lu
215cd27b37
add authorizing fileId write access
...
need to secure upload/update/delete for benchmark/filer/mount
need to add secure grpc
2019-02-14 00:08:20 -08:00
Chris Lu
4ff4a147b2
cleanup security.Secret
2019-02-09 21:56:32 -08:00
Chris Lu
501bd72b1c
wip: add security.toml file
2019-02-09 21:07:12 -08:00
Sergey
aa5ccff6d2
fixing of typos
2019-02-06 18:59:15 +05:00
Chris Lu
a6bffc58f9
go fmt
2019-01-18 14:15:10 -08:00
Chris Lu
67e2ea72be
master add separate grpc port
...
due to https://github.com/soheilhy/cmux/issues/64
fix https://github.com/chrislusf/seaweedfs/issues/820
fix https://github.com/chrislusf/seaweedfs/issues/840
fix https://github.com/chrislusf/seaweedfs/issues/841
2019-01-18 14:14:47 -08:00
bingoohuang
ab6be025d7
go fmt and fix some typo
2019-01-17 09:17:19 +08:00
bingoohuang
6ddfaf33cb
extract VolumeFileScanner for ScanVolumeFile
2019-01-16 17:48:59 +08:00
Chris Lu
6740427638
weed mount: reduce default chunk size
2019-01-05 15:17:03 -08:00
Chris Lu
55f7d8f801
remove extra log
2018-12-29 18:00:19 -08:00
Chris Lu
170ed53da5
weed mount: mac add AutoXattr
2018-12-29 13:55:38 -08:00
Chris Lu
9f3836072a
weed mount: no mac finder broswe mode
2018-12-29 13:16:23 -08:00
Chris Lu
d7bf6c5b91
switching_to_seaweedfs_fuse
2018-12-29 10:31:36 -08:00
Chris Lu
03c7447ad6
set mount point to existing permissions
...
fix https://github.com/chrislusf/seaweedfs/issues/806
2018-12-28 23:36:13 -08:00
Chris Lu
2a19511ba8
avoid gzip benchmark data
...
related to https://github.com/chrislusf/seaweedfs/issues/804
2018-12-27 13:10:46 -08:00
Chris Lu
ba923e2864
error if the last arguement does not end with forward slash
2018-12-23 18:20:11 -08:00
Chris Lu
49f40cd145
adjust help message
2018-12-22 15:05:15 -08:00
Chris Lu
852ee21835
avoid .gz auto decompression
2018-12-22 13:58:16 -08:00
Chris Lu
93d0ad09af
error log format
2018-12-16 17:37:10 -08:00
Chris Lu
c28e8a2397
refactoring
2018-12-06 00:44:41 -08:00
Chris Lu
ffa2827ab1
fail fast if two notification queues or inputs are enabled
2018-12-06 00:37:59 -08:00
Chris Lu
52b24a9902
default "weed server -filer" to same directory as -mdir
2018-12-05 23:24:25 -08:00
Chris Lu
fb43a9d7b9
adjust message
2018-12-05 02:03:03 -08:00
Chris Lu
1cbd53c01c
WIP SeaweedFileSystem added mkdirs, getFileStatus, listStatus, delete
2018-11-25 13:43:26 -08:00
Chris Lu
86dd933596
go fmt
2018-11-23 00:26:15 -08:00
Chris Lu
6344469a9b
set fuse subtype
2018-11-22 23:04:23 -08:00
Chris Lu
60d3681371
default images.fix.orientation to false
2018-11-21 16:50:42 -08:00
Chris Lu
c9b3ef58de
options to control filer.export verbosity
2018-11-21 15:15:40 -08:00
Chris Lu
2f2e417b68
detailed export
2018-11-21 15:12:47 -08:00
Chris Lu
2e6a3c7b16
reduce memory usage
2018-11-14 22:48:54 -08:00
Chris Lu
a62adf62ca
a bit faster
2018-11-12 11:19:13 -08:00
Chris Lu
affe708961
a bit faster
2018-11-11 00:43:30 -08:00
Chris Lu
a09eaf8756
adjusting, seems there are some perf benefit
2018-11-11 00:08:22 -08:00
Chris Lu
cbd94b18a5
improve "ls -al" performance for large directory
2018-11-08 07:37:34 -08:00
Chris Lu
85d1d99459
default "weed export to current enabled store"
2018-11-04 12:59:29 -08:00
Chris Lu
6c45787a37
selectively export files
2018-11-04 12:07:33 -08:00
Chris Lu
ff4189ab25
go fmt
2018-11-04 11:59:08 -08:00
Chris Lu
8dfac6a4cf
working b2 sink
2018-11-04 11:58:59 -08:00
Chris Lu
a64613172d
bootstrap filer replication with weed filer.export -targetStore=notification
2018-11-03 12:43:45 -07:00
Chris Lu
3674ad9f8e
go fmt
2018-11-01 01:12:21 -07:00
Chris Lu
db584ff7f8
separate into notification.toml, add gcp pub/sub message queue
2018-11-01 01:11:09 -07:00
Chris Lu
4c97ff3717
support AWS SQS as file change notification message queue
2018-10-31 01:11:19 -07:00
Chris Lu
fe31f389b0
add configuration instructions
2018-10-30 02:29:11 -07:00
Chris Lu
c58b73ad9b
adding BackBlaze, Azure
2018-10-23 23:59:40 -07:00
Chris Lu
fda771c83f
migrate volume sync status to grpc API on volume server
2018-10-15 01:19:15 -07:00
Chris Lu
b1daede91b
move volume vacuum to gRpc
2018-10-14 23:12:43 -07:00
Chris Lu
556382ff5f
adding basic grpc to volume server
2018-10-11 01:16:33 -07:00
Chris Lu
08266b7256
go fmt
2018-10-11 00:08:13 -07:00
Chris Lu
1d7e01b754
reformat
2018-10-11 00:05:54 -07:00
Chris Lu
da6154b29c
refactor volume server to startVolumeServer()
2018-10-11 00:04:31 -07:00
Chris Lu
60d2f1557d
refactor func name
2018-10-10 23:19:54 -07:00
Chris Lu
9f4c2f87d3
adding filer replication sink to Azure
2018-10-09 01:35:48 -07:00
Chris Lu
96a23591e7
remove comments
2018-10-07 22:21:00 -07:00
Chris Lu
a5b5887efc
add filer replication to aws s3
2018-10-06 17:10:15 -07:00
Chris Lu
84d4a2c8a0
add filer replication to google cloud storage
2018-10-06 13:08:38 -07:00
Chris Lu
14858f6224
add logging
2018-10-06 13:01:38 -07:00
Chris Lu
56c5c7b1b6
add google cloud storage
2018-10-04 01:14:44 -07:00
Chris Lu
e8ef501f02
add s3 replication sink
2018-10-03 23:36:52 -07:00
Chris Lu
98384b62fd
adjust description
2018-09-28 01:58:34 -07:00
Chris Lu
12ba8b9799
generate useable filer.toml
2018-09-23 09:26:25 -07:00
Chris Lu
b16d47030e
save and use last processed kafka offsets
2018-09-23 01:34:40 -07:00
Chris Lu
9fe24991d5
refactoring
2018-09-23 00:40:36 -07:00
Chris Lu
7d6b2a4740
add ETag to upload results and chunks
2018-09-22 22:12:21 -07:00
Chris Lu
420f068376
add more instructions
2018-09-22 22:12:06 -07:00
Chris Lu
01ceace18e
adjust sink options
2018-09-22 00:53:52 -07:00
Chris Lu
402ab598b6
go fmt
2018-09-22 00:12:10 -07:00
Chris Lu
7bb62b9f21
delay file creation without chunks from dir.Create to file.Flush
2018-09-22 00:11:46 -07:00
Chris Lu
db69ce89f0
go fmt
2018-09-21 01:56:43 -07:00
Chris Lu
a6cfaba018
able to sync the changes
2018-09-21 01:54:29 -07:00
Chris Lu
a8cd7df44a
let filer choose default replication
2018-09-20 09:00:15 -07:00
Chris Lu
d5149e592e
add logging on kafka reading
2018-09-19 22:31:45 -07:00
Chris Lu
25fb6f9a46
fix compilation
2018-09-17 02:23:21 -07:00
Chris Lu
779641e9d4
adjust replicated entry name
2018-09-17 01:37:24 -07:00
Chris Lu
b7a375f5aa
set large limit for "weed filer.export" directory listing
2018-09-17 00:42:36 -07:00
Chris Lu
4e1f1a0b92
set large limit for "weed mount" directory listing
2018-09-17 00:41:33 -07:00
Chris Lu
788acdf527
add WIP filer.replicate
2018-09-17 00:27:56 -07:00
Chris Lu
7ad5a90127
go fmt
2018-09-03 16:47:00 -07:00
Chris Lu
6e3f4d1079
add working filer.export command
2018-08-23 00:02:04 -07:00
Chris Lu
98b8f8649d
adding filer.export prototype
2018-08-22 23:54:22 -07:00
Chris Lu
ae4e654d2a
add example configuration for default redis create-cluster setup
2018-08-22 18:23:19 -07:00
Chris Lu
748870b03f
add "weed scaffold" command
2018-08-19 15:36:30 -07:00
Chris Lu
f827ada811
merge notification config with filer.toml
2018-08-19 15:17:55 -07:00
Chris Lu
f036ef8a3c
add filer notification
2018-08-13 01:20:49 -07:00
Chris Lu
75d63db60d
randomize raft server startup
...
also some go fmt
2018-08-12 14:27:14 -07:00
Chris Lu
eca4b928d2
ensure master server count is odd
2018-08-12 14:25:31 -07:00
Chris Lu
2e427c66ac
small refactoring
2018-08-10 23:47:17 -07:00
Chris Lu
44d8e35988
add listener for volumd id location for benchmark tool
2018-07-31 19:12:36 -07:00
Chris Lu
d09606c65b
refactoring
2018-07-29 02:25:24 -07:00
Chris Lu
4f317c7e3d
working version3
2018-07-24 01:36:04 -07:00
Chris Lu
7f32eb1e25
s3 add https support
2018-07-22 21:28:54 -07:00
Chris Lu
056871242b
remove println
2018-07-22 01:17:49 -07:00
Chris Lu
0655151b8c
go fmt
2018-07-22 01:15:11 -07:00
Chris Lu
6319d84f42
s3 API add ListObjectsV1
2018-07-22 01:14:36 -07:00
Chris Lu
922c614bde
fix compilation problem
2018-07-21 20:42:14 -07:00
Chris Lu
16c66e097b
adjust default buckets folder
2018-07-21 20:29:33 -07:00
Chris Lu
7e2031b18f
go fmt
2018-07-21 17:39:10 -07:00
Chris Lu
c7228fefa7
add bucket creation and deletion
...
1. option for "weed s3 -filer.dir.buckets" to choose a folder for buckets
2. create a bucket
3. delete a bucket, recursively delete all metadata on filer
2018-07-19 01:21:44 -07:00
Chris Lu
834a25f084
add list all my buckets
2018-07-18 02:37:09 -07:00
Chris Lu
7abfab8e77
add feature to mount a specific filer path to local directory
2018-07-14 21:09:21 -07:00
Chris Lu
3edfe1d28f
extend export command to show tombstone + change output format to CSV
...
merging https://github.com/chrislusf/seaweedfs/pull/610 and add "-limit" option
2018-07-14 20:51:17 -07:00
Chris Lu
6d58916cfd
fix compilation error
2018-07-14 13:39:57 -07:00
Chris Lu
842dab07b4
weed mount can request to prioritize to write to a data center
2018-07-14 13:36:28 -07:00
Chris Lu
1e1c368ae8
adjust filer help message
2018-07-10 00:22:11 -07:00
Chris Lu
50d351944f
filer add data center option
2018-07-09 02:22:48 -07:00
Chris Lu
d4d7ced922
refactoring: add type for needle id, offset
...
later the type size can possibly be adjusted
2018-07-08 02:28:04 -07:00
Chris Lu
922032b9bb
add filer sub dir list size limit to "weed server"
2018-07-08 02:11:36 -07:00
Chris Lu
702ff48cda
add dir list limit option
2018-07-07 02:18:47 -07:00
Chris Lu
2ad45ca04f
refactor
2018-07-07 02:03:25 -07:00
Chris Lu
77fc8c5914
keep alive for gRpc calls
2018-07-03 19:07:55 -07:00
Chris Lu
5bd72696ac
weed mount add ttl option
2018-06-11 23:13:33 -07:00
Chris Lu
98110c1697
add collection, replication, ttl to each file
2018-06-10 16:57:32 -07:00
Chris Lu
299312c805
use separate filer grpc port
2018-06-05 23:37:41 -07:00
Chris Lu
43e3f5724c
use fixed list of masters in both filer and volume servers
2018-06-01 00:39:39 -07:00
Chris Lu
dc13e10637
fix copy error on 0 size files
2018-05-30 22:28:14 -07:00
Chris Lu
44acf4b756
fix file name
...
f.Name() is not the base file name
2018-05-30 20:48:13 -07:00
Chris Lu
0301504184
add mime, use simple insert and update filer store API
...
1. add mime type to file in filer
2. purge old chunks if overwrite during insert
2018-05-30 20:24:57 -07:00
Chris Lu
26e7cd8c75
assign a different volume on large file copying
2018-05-30 01:05:26 -07:00
Chris Lu
2fe0d479f1
filer.copy supports chunking
2018-05-30 00:54:56 -07:00
Chris Lu
2d7dea1a09
a little refactoring
2018-05-29 23:52:27 -07:00
Chris Lu
8a48baa056
add single chunk file copying to new filer
2018-05-29 23:46:45 -07:00
Chris Lu
4e3ea49cff
properly working local write buffer
2018-05-29 01:21:21 -07:00
Chris Lu
5c4480ec6c
add mountOptions.chunkSizeLimitMB, remove cmdMount.IsDebug
2018-05-28 13:42:25 -07:00
Chris Lu
8ab7dd9d08
weed mount add options for collection and replication
2018-05-28 13:24:48 -07:00
Chris Lu
458ada173e
go fmt
2018-05-27 11:52:26 -07:00
Chris Lu
9e77563c99
add leveldb store
...
1. switch to viper for filer store configuration
2. simplify FindEntry() return values, removing “found”
3. add leveldb store
2018-05-26 03:49:46 -07:00
Chris Lu
9dd228747c
filer copy added uid/gid
2018-05-21 01:25:30 -07:00
Chris Lu
c5cf9bd290
properly working filer
2018-05-13 23:56:16 -07:00
Chris Lu
9f345da20f
mv filer proto to filer_pb
2018-05-09 23:18:02 -07:00
Chris Lu
942c2cbd7b
mv pb to master_pb
2018-05-09 23:11:54 -07:00
Chris Lu
43a69d20bf
change filer API to gRPC
2018-05-08 01:59:43 -07:00
Chris Lu
4936d6c342
adjusting for FUSE
2018-05-07 00:37:47 -07:00
Chris Lu
456738ba64
refactoring fuse
2018-05-05 22:47:16 -07:00
Chris Lu
fffb14bc87
better support FUSE Lookup()
2018-05-05 02:01:50 -07:00
Chris Lu
4b9b9547d4
unused option
2018-02-27 13:57:15 -08:00
Chris Lu
61ff330e99
adjust instructions
2018-01-15 20:33:12 -08:00
Chris Lu
ef6224e45d
filer default to auto chunking file size to 32MB
2017-09-15 08:24:30 -07:00
Chris Lu
58344980e4
preallocate disk space during compaction also, add cleanup for failed compaction
2017-08-29 23:59:53 -07:00
Chris Lu
dbee816205
fix typo
...
fix https://github.com/chrislusf/seaweedfs/issues/549
2017-08-10 10:26:39 -07:00
Chris Lu
e0dfa7a069
ensure 30GB or less
...
avoid https://github.com/chrislusf/seaweedfs/issues/535
2017-07-16 21:40:55 -07:00
Chris Lu
1fbb8723dc
add cpu and mem pprof
2017-06-22 01:33:58 -07:00
Chris Lu
5047bdb4a2
skip bytes cache
2017-06-03 11:44:24 -07:00
Chris Lu
478fe0ecf2
filer add readonly public port
2017-05-27 20:14:22 -07:00
Chris Lu
7ecc0f4b11
rename variables
2017-05-27 18:11:18 -07:00
Chris Lu
82c3ccc8dd
add btree for volume index
2017-05-26 22:51:25 -07:00
Chris Lu
4ce6586710
use local random generator to avoid global lock
2017-05-19 23:33:36 -07:00
sparklxb
c46e91d229
complement weed mount: add read and delete
2017-03-17 15:02:27 +08:00
sparklxb
3a67b6ddc1
complement weed mount
2017-03-10 14:46:00 +08:00
sparklxb
7f85e15990
complement weed mount
2017-03-10 10:28:19 +08:00
sparklxb
813a4f7456
Fixes #266
2017-03-06 13:04:52 +08:00
Chris Lu
41f686ab9d
remove weeds.conf
2017-02-12 21:58:44 -08:00
Chris Lu
d953ed442c
auto detect volume server ip address
2017-01-12 13:42:53 -08:00
Chris Lu
1be354bdce
default IP to "localhost"
...
so that the master can know the ip address.
2017-01-10 23:57:59 -08:00
Chris Lu
e767c3ea4f
disable master connection timeout
...
temporarily disable master connection timeout due to heartbeat
connection timeout
2017-01-10 01:30:00 -08:00
Chris Lu
e46c3415f7
gRpc for master~volume heartbeat
2017-01-10 01:01:12 -08:00
sparklxb
66e7013dfe
suport uploading files to specific dataCenter
2017-01-09 23:34:47 +08:00
Chris Lu
53cf1b4900
change the basic unit to MB
2017-01-08 14:57:32 -08:00
Chris Lu
ed44f12f6d
support Fallocate on linux
2017-01-08 11:01:46 -08:00
Chris Lu
13e7069eb9
keep track of total data file size
2017-01-06 10:22:20 -08:00
lixianbin
d96d0a87cf
fix bug: upload big .gz file more than maxMB
2017-01-04 11:23:40 +08:00
Chris Lu
dcaf1796fe
add option to enable caching
2016-11-16 07:09:57 -08:00
Chris Lu
1bc041b46d
add a new way to manually compact corrupted volume
...
fix https://github.com/chrislusf/seaweedfs/issues/371
2016-09-22 20:31:20 -07:00
霍晓栋
1c2affea8e
Merge remote-tracking branch 'upstream/master'
2016-09-08 10:34:24 +08:00
Chris Lu
c4b7966dbe
minor help message change
2016-09-07 18:21:05 -07:00
霍晓栋
b0035747e3
add filer support
2016-08-31 11:32:30 +08:00
Mike Tolman
01d3f69c52
Adding AutoChunk/MaxMB Support to Filer API
...
This is related to the following issue I added to chrislusf/seaweedfs:
https://github.com/chrislusf/seaweedfs/issues/342
2016-08-05 16:01:30 -06:00
Mike Tolman
a89a3c86d0
Revert "Add AutoChunking to the Filer API, so that you can upload really large files through the filer API."
...
This reverts commit 09059bfdcc
.
2016-08-05 15:47:46 -06:00
Mike Tolman
ce99bb927d
Revert "Adding HTTP verb whitelisting options."
...
This reverts commit 34837afc7a
.
2016-08-05 15:45:48 -06:00
Mike Tolman
34837afc7a
Adding HTTP verb whitelisting options.
2016-08-05 15:23:43 -06:00
Mike Tolman
09059bfdcc
Add AutoChunking to the Filer API, so that you can upload really large files through the filer API.
2016-08-05 15:19:10 -06:00
Chris Lu
185a916f5e
adjusting command options
2016-07-21 15:00:07 -07:00
Chris Lu
cdae9fc680
add "weed copy" command to copy files to filer
2016-07-20 23:45:55 -07:00
Chris Lu
d3ae3a8e14
collect proof when stopping
2016-07-05 10:23:14 -07:00
Chris Lu
b38ff39e8b
add master profiling option
2016-07-03 15:39:57 -07:00
霍晓栋
1f63094542
refactor Volume Assign function
2016-06-26 10:50:18 +08:00
Chris Lu
5ce6bbf076
directory structure change to work with glide
...
glide has its own requirements. My previous workaround caused me some
code checkin errors. Need to fix this.
2016-06-02 18:09:14 -07:00