Chris Lu
7764e0465c
refactoring
2020-04-12 21:00:55 -07:00
Chris Lu
8e23dc078b
refactoring
2020-04-12 20:48:21 -07:00
Chris Lu
6f948e4887
remove configurable topics folder location
2020-04-12 13:07:59 -07:00
Chris Lu
e909b55633
refactoring
2020-04-11 12:39:46 -07:00
Chris Lu
ec2eb8bc48
add If-None-Match and If-Modified-Since
...
fix https://github.com/chrislusf/seaweedfs/issues/1269
2020-04-08 08:12:00 -07:00
Chris Lu
5a5908407d
filer: support larger file size
...
fix https://github.com/chrislusf/seaweedfs/issues/1257
2020-04-05 21:48:45 -07:00
Chris Lu
2d43f85577
watch entries with common path prefix
2020-04-05 12:51:21 -07:00
Chris Lu
dcc6b6a3ed
go fmt
2020-04-05 00:52:31 -07:00
Chris Lu
bf270d9e8c
filer: able to tail meta data changes
2020-04-05 00:51:16 -07:00
Chris Lu
50a5018b7f
writing meta logs is working
2020-03-30 01:19:33 -07:00
Chris Lu
a75d50bbb8
FUSE: add configurable in memory chunk cache size
2020-03-28 14:07:16 -07:00
Chris Lu
826bc0b7e3
FUSE: add chunk cache for recently accessed file chunks
2020-03-28 13:43:31 -07:00
Chris Lu
712720864d
error checking for remove
2020-03-26 02:21:30 -07:00
Chris Lu
dd5b582d05
go fmt
2020-03-26 00:09:01 -07:00
Chris Lu
6d87a0c5f5
batch delete orphan data by file keys
2020-03-25 02:21:15 -07:00
Chris Lu
7f1e3c843d
refactoring
2020-03-23 01:14:21 -07:00
Chris Lu
c07bcd5065
refactoring
2020-03-23 00:30:02 -07:00
Chris Lu
654a69ff52
refactoring
2020-03-23 00:06:24 -07:00
Chris Lu
c0f0fdb3ba
refactoring
2020-03-23 00:01:34 -07:00
Chris Lu
c4bea45099
S3 API: fix DeleteMultipleObjectsHandler
...
fix https://github.com/chrislusf/seaweedfs/issues/1241
2020-03-20 14:17:31 -07:00
Chris Lu
bec6ec7db6
go fmt
2020-03-17 10:01:55 -07:00
Chris Lu
c3cb6fa1d7
volume: compaction can cause readonly volumes
...
address https://github.com/chrislusf/seaweedfs/issues/1233
2020-03-17 09:43:57 -07:00
Chris Lu
2e3f6ad3a9
filer: remember content is gzipped or not
2020-03-08 21:39:33 -07:00
Chris Lu
afb20de14c
breaks dependency loop
2020-03-07 17:01:39 -08:00
Chris Lu
8645283a7b
fuse mount: avoid lookup nil entry
...
fix https://github.com/chrislusf/seaweedfs/issues/1221
2020-03-07 16:51:46 -08:00
Chris Lu
1ae83c2938
go fmt
2020-03-07 06:12:57 -08:00
Chris Lu
13e215ee5c
filer: option to encrypt data on volume server
2020-03-06 00:49:47 -08:00
Chris Lu
f90c43635d
refactoring
2020-03-04 00:39:47 -08:00
Chris Lu
410bce3925
go fmt
2020-03-01 22:39:08 -08:00
Chris Lu
6a8484b4ae
master able to list all master clients by type
2020-03-01 22:13:47 -08:00
Chris Lu
0ca68a2a6d
WIP
2020-03-01 22:13:15 -08:00
Chris Lu
555413d9fc
weed queue starts
2020-02-27 00:07:13 -08:00
Chris Lu
892e726eb9
avoid reusing context object
...
fix https://github.com/chrislusf/seaweedfs/issues/1182
2020-02-25 21:50:12 -08:00
Chris Lu
bd3254b53f
adjust logging
2020-02-25 17:24:08 -08:00
Chris Lu
0841bedb15
move filer assign volume grpc errror to response
2020-02-25 17:15:09 -08:00
Chris Lu
bc38b72a20
s3: implemented DeleteMultipleObjects
2020-02-25 14:38:36 -08:00
Chris Lu
6ab7368ef2
filer: dynamically create bucket under /buckets folder
2020-02-24 22:28:45 -08:00
Chris Lu
d8dec2323b
s3: move buckets folder configuration to filer
2020-02-24 14:34:14 -08:00
Chris Lu
e96f85ed8a
make pb files
2020-02-23 18:06:34 -08:00
Chris Lu
346a1cf0b9
Merge pull request #1207 from LazyDBA247-Anyvision/update-stats-and-status-volume-server
...
Update stats and status volume server
2020-02-23 18:05:25 -08:00
Chris Lu
bbb163c5ad
add for https://github.com/chrislusf/seaweedfs/pull/1207
2020-02-23 18:04:31 -08:00
LazyDBA247-Anyvision
ec8619176a
volume server protobuf: add precentFree & percentUsage to the DiskStatus
2020-02-23 23:24:24 +02:00
Chris Lu
f55f49970f
move volume server stats to grpc
2020-02-21 21:45:03 -08:00
Chris Lu
0a57db7026
Merge pull request #1200 from chrislusf/add_s3_acl
...
Add s3 ACL
2020-02-17 12:53:30 -08:00
Chris Lu
4cdde5f569
configuration stores the identity list
2020-02-17 12:31:59 -08:00
Chris Lu
9b6296e77a
volume: add grpc file read operation
...
This is added more for performance benchmarking.
2020-02-14 00:37:32 -08:00
Chris Lu
9ed364f053
support acl
2020-02-09 14:30:02 -08:00
Chris Lu
40ae533fa3
shell: add volume.configure.replication to change replication for a volume
...
fix https://github.com/chrislusf/seaweedfs/issues/1192
2020-02-02 15:37:23 -08:00
Chris Lu
c48fc8b4de
grpc send error via response instead of grpc error
2020-01-25 09:17:19 -08:00
Chris Lu
d4e75a0d18
filer: option to create file only if it is new, O_EXCL
2020-01-22 11:42:40 -08:00
Chris Lu
03ab0b1b9a
correct .vif file creation
2019-12-28 21:40:12 -08:00
Chris Lu
0cc8347a3a
go fmt
2019-12-28 21:13:10 -08:00
Chris Lu
37b64a50b4
ec: generate and copy .vif file
2019-12-28 12:44:59 -08:00
Chris Lu
c06f7eb48a
load volume info from .vif file, use superblock as a backup
2019-12-28 12:28:58 -08:00
Chris Lu
2000284435
rename volume tier info to volume info
2019-12-28 11:21:49 -08:00
Chris Lu
b3c83f9404
fix spelling
2019-12-28 11:16:10 -08:00
Chris Lu
48d28d3eb2
tier: support remote file attributes and remember the file extension
2019-12-25 21:37:24 -08:00
Chris Lu
d960b3474a
tier storage: support downloading the remote dat files
2019-12-25 09:53:13 -08:00
Chris Lu
09ca936c78
shell: add ec.decode command
2019-12-23 12:48:20 -08:00
Chris Lu
0fa1269bc7
filer: streaming file listing
2019-12-13 00:22:37 -08:00
Chris Lu
e426bd541e
grow volumes on volume servers with slots freed by cloud storage
2019-12-03 21:36:42 -08:00
Chris Lu
1becbce657
display remote volumes on volume server ui page
2019-12-02 23:23:54 -08:00
Chris Lu
70648d35ad
go fmt
2019-12-02 20:49:58 -08:00
Chris Lu
126fb492d2
fix test
2019-12-02 15:52:33 -08:00
Chris Lu
ec8de250e2
tiered storage: can copy to s3, read from s3
...
master not aware tiered volume yet, file assigning is not working yet
2019-12-02 15:08:31 -08:00
Chris Lu
228231f3d7
minor
2019-12-02 15:08:31 -08:00
Chris Lu
0da7b894cc
pass backend config from master to volume servers
2019-11-29 01:05:09 -08:00
Chris Lu
586798ecc0
go fmt
2019-11-27 12:34:57 -08:00
Chris Lu
0f9ba84274
s3
2019-11-27 03:09:45 -08:00
Chris Lu
471902cd51
change parameter name
2019-11-10 12:05:45 -08:00
zhangsong
61fa485700
add volume number param in assign operation
2019-11-10 20:11:03 +08:00
Chris Lu
faec9076a4
adjust parameter names
2019-10-21 22:57:01 -07:00
joeslay
d53aee179b
Merge pull request #10 from chrislusf/master
...
merge seaweed master
2019-10-14 16:03:40 +01:00
Chris Lu
f8d4b7d1c0
support basic json filtering and selection
2019-10-06 22:35:08 -07:00
Chris Lu
cf47f657af
scaffold for volume server query feature
2019-10-06 22:35:08 -07:00
j.laycock
6fc6322c90
Change joeslay paths to chrislusf paths
2019-09-12 14:18:21 +01:00
joeslay
d8c34b032f
Merge pull request #6 from chrislusf/master
...
merge seaweed master
2019-09-12 11:44:09 +01:00
Chris Lu
ae53f63680
filer: recursive deletion optionally ignoring any errors
...
fix https://github.com/chrislusf/seaweedfs/issues/1062
2019-09-11 20:26:20 -07:00
j.laycock
84f2dc3b53
Remove test code
2019-09-05 11:14:02 +01:00
Tom Maxwell
4a878c0006
Changed the InMemory bool to a uint32 so that it can be used to alter how much space to reserve
2019-09-04 15:27:14 +01:00
j.laycock
9aa0859697
Remove inmemory from vaccum stuff
2019-09-03 18:20:45 +01:00
j.laycock
cbd0a98fa1
Add InMemory to Volume Grow structure
2019-09-03 18:19:02 +01:00
Tom Maxwell
d637d86d22
Changes to try and pass the URL parameters through - in memory flag not working still
2019-09-03 15:41:28 +01:00
j.laycock
595a1beff0
Swap imports to use joeslay
2019-09-02 11:28:40 +01:00
Chris Lu
19360ab10a
master: redirect clients to the new leader
2019-07-31 01:54:42 -07:00
Chris Lu
6883f9e322
mark volume readonly before ec encoding
2019-06-26 23:02:22 -07:00
Chris Lu
d5560f2705
simplify metrics settings
2019-06-23 15:29:49 -07:00
Chris Lu
8da5d5b094
filer.copy: use filer settings, avoid unnecessary command line options
...
fix https://github.com/chrislusf/seaweedfs/issues/968
2019-06-23 01:57:35 -07:00
Chris Lu
1babec00e7
check deleted chunks faster
2019-06-22 13:22:22 -07:00
Chris Lu
cd45ab072a
fix compilation error
2019-06-22 12:30:08 -07:00
Chris Lu
a111f26fe6
avoid nil
...
fix https://github.com/chrislusf/seaweedfs/issues/988
2019-06-21 20:56:27 -07:00
Chris Lu
059ef879a8
fix issue 986
...
fix issue 986
2019-06-21 13:06:04 -07:00
Chris Lu
f88a8bda7b
ec deletion works
2019-06-21 01:14:10 -07:00
Chris Lu
4cea8aefd0
add grpc VolumeEcBlobDelete
2019-06-20 00:17:11 -07:00
Chris Lu
d8ed73926d
volume servers get metrics address and interval from the master
2019-06-17 14:51:47 -07:00
Chris Lu
450f4733ce
report empty volume or ec shards
2019-06-05 13:32:33 -07:00
Chris Lu
7e80b2b882
fix multiple bugs
2019-06-03 02:26:31 -07:00
Chris Lu
f919d0235c
ec encode volumes quiet for a period of time
2019-05-31 23:41:17 -07:00
Chris Lu
1d111d6ce8
collection list normal and ec volumes
2019-05-30 09:27:23 -07:00
Chris Lu
3f9ecee40f
working with reading remote intervals
2019-05-28 21:29:07 -07:00
Chris Lu
2858a954b3
read ec volume shards locations from master
2019-05-28 00:13:13 -07:00
Chris Lu
b4b407e403
add grpc ec shard read
2019-05-27 11:59:03 -07:00
Chris Lu
db94a41f9e
mount/unmount ec shards
2019-05-25 23:23:19 -07:00
Chris Lu
41e8ae61f8
generate, balance, delete copied shards, delete old volume
2019-05-25 14:02:06 -07:00
Chris Lu
6f4b09b6a4
pb shard info uses ShardBits instead one message for one shard
2019-05-24 13:28:44 -07:00
Chris Lu
7180520889
ec shard info can be queried via VolumeList()
2019-05-22 22:44:28 -07:00
Chris Lu
17ac1290c0
volume: load ec shards during heartbeats to master
2019-05-21 22:41:20 -07:00
Chris Lu
fbbc74abb4
adds VolumeEcGenerateSlices, VolumeEcCopy
2019-05-20 00:53:17 -07:00
Chris Lu
698dea779c
refactoring
2019-05-19 21:37:49 -07:00
Chris Lu
bf9d490538
fix for tests
2019-05-17 02:28:20 -07:00
Chris Lu
82b0759493
filer: migrating filer store from persisting shorter structured file id instead of a string
2019-05-17 02:03:23 -07:00
Chris Lu
3ff92037cc
shell: add volume.balance command
2019-05-05 21:17:23 -07:00
Chris Lu
316bd27f75
add remove volumes with version info
2019-04-20 23:53:37 -07:00
Chris Lu
2ffe98443b
refactoring
2019-04-20 12:05:28 -07:00
Chris Lu
6fc1f53018
shell: add command volume.move
2019-04-20 11:35:23 -07:00
Chris Lu
3b3651dea3
volume: atomic copying file, adds version and stopOffset
2019-04-19 12:29:49 -07:00
Chris Lu
ac2727853f
fix needle map entry size
2019-04-19 00:39:34 -07:00
Chris Lu
fa176fe80f
volume tailing chunks large files
2019-04-18 19:22:13 -07:00
Chris Lu
3dce1016cb
add volume tailer
2019-04-18 11:05:02 -07:00
Chris Lu
b09e8dbb37
add VolumeStreamFollow, but not used yet
2019-04-18 00:18:29 -07:00
Chris Lu
13ad5c1966
refactoring
2019-04-17 22:04:49 -07:00
Chris Lu
b3b42bc947
replicate need to include new entry path
2019-04-16 00:44:31 -07:00
Chris Lu
cb2061f86a
Merge pull request #925 from stlpmo-jn/checkUnhealthReplication
...
the implement of repair the error replications of the volume
2019-04-10 23:37:12 -07:00
Chris Lu
2f76681d62
refactor: adjust grpc API name
2019-04-10 21:41:17 -07:00
stlpmo-jn
c1a0403da2
repair the error replications of the volume
2019-04-10 19:41:55 +08:00
Chris Lu
97406333a5
support atomic renaming for mysql/postgres filer store
2019-03-30 23:08:29 -07:00
Chris Lu
19728fe3f6
remove deprecated code
2019-03-25 23:12:14 -07:00
Chris Lu
70815e9124
WIP
2019-03-25 09:16:12 -07:00
Chris Lu
95e0520182
weed volume: add grpc operation to relicate a volume to local
2019-03-23 11:33:34 -07:00
Chris Lu
ece9d13312
volume info collect compact revision number
2019-03-18 09:32:21 -07:00
Chris Lu
aca653c08b
weed shell: list volumes
2019-03-17 20:27:08 -07:00
Chris Lu
657dd2e6c9
add shell command to list all collections
2019-03-16 13:43:16 -07:00
Chris Lu
74fb237727
benchmark can work in secure mode
2019-02-15 00:09:19 -08:00
Chris Lu
4ff4a147b2
cleanup security.Secret
2019-02-09 21:56:32 -08:00
Chris Lu
0c932d1738
add volume delete grpc API
...
fix https://github.com/chrislusf/seaweedfs/issues/808
2018-12-29 00:03:30 -08:00
Chris Lu
253f190f48
weed mount add symlink support
2018-12-25 22:45:44 -08:00
Chris Lu
5333f2984a
streaming updates for large entries or large index file
...
fix https://github.com/chrislusf/seaweedfs/issues/801
2018-12-22 11:10:08 -08:00
Chris Lu
39ea8484c1
adjust proto API
2018-12-11 07:17:10 -08:00
Chris Lu
1cbd53c01c
WIP SeaweedFileSystem added mkdirs, getFileStatus, listStatus, delete
2018-11-25 13:43:26 -08:00
Chris Lu
444dfded84
add fs.FSStatfser for SeaweedFS weed mount
2018-11-23 00:24:51 -08:00
Chris Lu
5065d4ab2d
master add grpc API for fileid assigning
2018-11-20 11:35:45 -08:00
Chris Lu
46eb77f9bb
move DiskStatus and MemStatus to protobuf
2018-10-15 22:25:28 -07:00
Chris Lu
eec951cad2
migrate volume sync to gRpc
2018-10-15 21:44:41 -07:00
Chris Lu
f8b2d3cacc
move volume mount/unmount on volume server to grpc
2018-10-15 01:48:15 -07:00
Chris Lu
fda771c83f
migrate volume sync status to grpc API on volume server
2018-10-15 01:19:15 -07:00
Chris Lu
b423bb9e2d
migrate assign volume to grpc API on volume server
2018-10-15 00:40:46 -07:00
Chris Lu
8301519fb0
migrate delete collection to grpc API on volume server
2018-10-15 00:03:55 -07:00
Chris Lu
b1daede91b
move volume vacuum to gRpc
2018-10-14 23:12:43 -07:00
Chris Lu
ff66269b62
use grpc to replace http APIs for batch volume id lookup and batch delete
...
1. remove batch volume id lookup http API /vol/lookup
2. remove batch delete http API /delete
2018-10-14 00:12:28 -07:00
Chris Lu
556382ff5f
adding basic grpc to volume server
2018-10-11 01:16:33 -07:00
Chris Lu
64ed47502b
rename file
2018-10-11 00:21:32 -07:00
Chris Lu
60c1ada4c1
remove filer grpc API GetEntryAttributes
2018-09-25 09:18:06 -07:00
Chris Lu
788acdf527
add WIP filer.replicate
2018-09-17 00:27:56 -07:00
Chris Lu
164091c269
add s3 multipart upload
2018-09-09 16:25:43 -07:00
Chris Lu
dd37f0540f
add extended attributes
2018-09-07 13:12:52 -07:00
Chris Lu
ce6a2140a2
preparing to support S3 multipart uploads
2018-09-04 00:42:44 -07:00
Chris Lu
76cbe8bf33
instant notification of new volumes added or deleted
2018-08-24 01:26:56 -07:00
Chris Lu
f036ef8a3c
add filer notification
2018-08-13 01:20:49 -07:00
Chris Lu
a12c7b86b0
broadcast messages of new and deleted volumes
2018-07-27 23:09:55 -07:00
Chris Lu
0d98949199
tmp commit
2018-07-27 01:54:45 -07:00
Chris Lu
6319d84f42
s3 API add ListObjectsV1
2018-07-22 01:14:36 -07:00
Chris Lu
568132c0c9
delete bucket also removes collection data
2018-07-20 00:10:01 -07:00
Chris Lu
c7228fefa7
add bucket creation and deletion
...
1. option for "weed s3 -filer.dir.buckets" to choose a folder for buckets
2. create a bucket
3. delete a bucket, recursively delete all metadata on filer
2018-07-19 01:21:44 -07:00
Chris Lu
842dab07b4
weed mount can request to prioritize to write to a data center
2018-07-14 13:36:28 -07:00
Chris Lu
28e5f20c8e
add super block extra for erasure coding
2018-06-28 22:20:37 -07:00
Chris Lu
f339e9c284
add support for extra super block info
2018-06-24 15:19:57 -07:00
Chris Lu
5bd72696ac
weed mount add ttl option
2018-06-11 23:13:33 -07:00
Chris Lu
98110c1697
add collection, replication, ttl to each file
2018-06-10 16:57:32 -07:00
Chris Lu
5c25d29272
support renaming files
2018-06-06 22:11:01 -07:00
Chris Lu
43e3f5724c
use fixed list of masters in both filer and volume servers
2018-06-01 00:39:39 -07:00
Chris Lu
0301504184
add mime, use simple insert and update filer store API
...
1. add mime type to file in filer
2. purge old chunks if overwrite during insert
2018-05-30 20:24:57 -07:00
Chris Lu
6de84c64c6
adding create time
2018-05-25 23:26:40 -07:00
Chris Lu
d773e11c7a
file handler directly read from volume servers
...
this mostly works fine now!
next: need to cache files to local disk
2018-05-24 01:22:37 -07:00
Chris Lu
873868cc10
not working now
...
need to add file handler
2018-05-22 04:31:44 -07:00
Chris Lu
7362de9a18
weed mount can work well
...
TODO: somehow filer url is returning empty content
2018-05-22 03:26:38 -07:00
Chris Lu
9dd228747c
filer copy added uid/gid
2018-05-21 01:25:30 -07:00
Chris Lu
f07482382b
able to update file content
...
having some issue when vi reports file changed.
2018-05-21 00:00:28 -07:00
Chris Lu
e31c514b00
adding modified time to file chunk
2018-05-16 00:54:44 -07:00
Chris Lu
b303a02461
cp file can work
...
1. consolidate to filer_pb.FileChunk
2. dir add file, mkdir
3. file flush, write
updates having issue
2018-05-16 00:08:44 -07:00
Chris Lu
9f345da20f
mv filer proto to filer_pb
2018-05-09 23:18:02 -07:00
Chris Lu
942c2cbd7b
mv pb to master_pb
2018-05-09 23:11:54 -07:00
Chris Lu
43a69d20bf
change filer API to gRPC
2018-05-08 01:59:43 -07:00
Chris Lu
5b0f5f456e
volume server get notified of leader change
2017-01-18 09:34:27 -08:00
Chris Lu
adc8d344a5
update proto buffer
2017-01-10 23:57:16 -08:00
Chris Lu
e46c3415f7
gRpc for master~volume heartbeat
2017-01-10 01:01:12 -08:00