mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-12-21 06:57:51 +08:00
Code Issues Packages Projects Releases Wiki Activity
11,170 Commits 32 Branches 296 Tags 152 MiB
34cdbdd279
Commit Graph

854 Commits

Author SHA1 Message Date
Eric Yang
ddd6bee970
ADHOC: Volume fsck use a time cutoff param (#3626) 
* ADHOC: cut off volumn fsck

* more

* fix typo

* add test

* modify name

* fix comment

* fix comments

* nit

* fix typo

* Update weed/shell/command_volume_fsck.go

Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
 2022-09-10 15:29:17 -07:00
Brian
4e3e2b1b82
Add option in volume.fix.replication to only fix under-replication and not delete volumes (#3640) 2022-09-10 08:05:28 -07:00
chrislu
fc4208d128 volume.balance: default to balance ALL_COLLECTIONS 2022-09-09 16:06:05 -07:00
chrislu
eb7cf3de81 a new test case 2022-09-08 14:04:42 -07:00
chrislu
4260804613 volume.balance: avoid moving out volume with max=1 2022-09-08 14:04:21 -07:00
chrislu
d653c5f811 unused 2022-09-08 12:27:53 -07:00
chrislu
03e22ddb43 rename 2022-09-08 12:06:03 -07:00
Konstantin Lebedev
916673ae24
do not print diskInfos with doVolumeCheckDisk in error log (#3598) 2022-09-05 07:49:26 -07:00
Brian
a28b668647
Added ability to change replication settings upon volume.tier.move (#3583) 2022-09-04 16:47:21 -07:00
chrislu
7eb121d5ae print process before do the work 2022-08-30 13:37:05 -07:00
askeipx
2e78a522ab
remove old raft servers if they don't answer to pings for too long (#3398) 
* remove old raft servers if they don't answer to pings for too long

add ping durations as options

rename ping fields

fix some todos

get masters through masterclient

raft remove server from leader

use raft servers to ping them

CheckMastersAlive for hashicorp raft only

* prepare blocking ping

* pass waitForReady as param

* pass waitForReady through all functions

* waitForReady works

* refactor

* remove unneeded params

* rollback unneeded changes

* fix
 2022-08-23 23:18:21 -07:00
chrislu
f0b4a7659a fix test 2022-08-23 01:52:29 -07:00
chrislu
bf88006037 format 2022-08-23 01:20:45 -07:00
famosss
911475526c
fix: TestCommandEcBalanceSmall Unit test fails when CommandEnv is nil (#3497) 2022-08-22 23:54:51 -07:00
chrislu
676e27c589 shell: stop long running jobs if lock is lost 2022-08-22 14:12:23 -07:00
chrislu
57e7582c36 refactoring 2022-08-22 14:11:13 -07:00
qzh
74b53729e1
feat(weed.move): add a speed limit parameter of moving files (#3478) 
* feat(weed.move): add a speed limit parameter of moving files

* fix(weed.move): set the default value of ioBytePerSecond to vs.compactionBytePerSecond

Co-authored-by: zhihao.qu <zhihao.qu@ly.com>
 2022-08-21 23:08:31 -07:00
chrislu
4573c99ae3 fix tests 2022-08-18 00:15:46 -07:00
chrislu
2b580a7566 also migrate jsonpb 2022-08-17 12:42:03 -07:00
chrislu
eaeb141b09 move proto package 2022-08-17 12:05:07 -07:00
chrislu
d3d52b1818 shell: fs.meta.load add quieter mode 
fix https://github.com/seaweedfs/seaweedfs/issues/3414
 2022-08-07 01:42:15 -07:00
chrislu
67814a5c79 refactor and fix strings.Split 2022-08-07 01:34:32 -07:00
Konstantin Lebedev
4d08393b7c
filer prefer volume server in same data center (#3405) 
* initial prefer same data center
https://github.com/seaweedfs/seaweedfs/issues/3404

* GetDataCenter

* prefer same data center for ReplicationSource

* GetDataCenterId

* remove glog
 2022-08-04 17:35:00 -07:00
chrislu
26dbc6c905 move to https://github.com/seaweedfs/seaweedfs 2022-07-29 00:17:28 -07:00
chrislu
a3fae7cd16 add an empty placeholder for mq.topic.list 2022-07-28 23:24:38 -07:00
chrislu
9f479aab98 allocate brokers to serve segments 2022-07-28 23:24:38 -07:00
chrislu
f25e273e32 display data center and rack in cluster.ps 2022-07-28 23:22:52 -07:00
chrislu
68065128b8 add dc and rack 2022-07-28 23:22:51 -07:00
chrislu
87ccef72db cluster.ps can list brokers 2022-07-28 23:22:07 -07:00
chrislu
7a6c559ab4 fix Change replication via volume.configure.replication by collection 
fix https://github.com/chrislusf/seaweedfs/issues/3346
 2022-07-21 22:01:05 -07:00
Chris Lu
4820b594a7
Merge pull request #3298 from kmlebedev/evacuateTargetServer 
volume server evacuate to target server
 2022-07-19 13:56:49 -07:00
Konstantin Lebedev
d3f7c09c03 remove ticker 
update the topology before each file
 2022-07-20 00:54:23 +05:00
Konstantin Lebedev
d422e7769c ticker.Stop 2022-07-18 16:38:19 +05:00
Konstantin Lebedev
73a0dea16b sync update topologyInfo 2022-07-18 16:27:02 +05:00
Konstantin Lebedev
2b4112e462 update otherNodes 2022-07-18 11:32:28 +05:00
Konstantin Lebedev
3c2774ec3d fix update topologyInfo 2022-07-18 01:46:31 +05:00
Konstantin Lebedev
4d5144e50d clouse background update 2022-07-15 13:51:08 +05:00
guosj
ab1b9697e6 supplement check duplicate accesskey 2022-07-13 17:28:20 +08:00
石昌林
392da4e038 Make the prompt information clearer 2022-07-13 11:39:45 +08:00
石昌林
ad5e8f68ec Check whether there is a duplicate accessKey when modifying iam 2022-07-13 11:36:11 +08:00
Konstantin Lebedev
6622240df7 fix TestVolumeServerEvacuate 2022-07-12 14:56:34 +05:00
Konstantin Lebedev
8372721a62 update topologyInfo 2022-07-12 13:47:21 +05:00
Konstantin Lebedev
ee95d23a22 help rack 2022-07-12 11:56:58 +05:00
Konstantin Lebedev
087fa1347f volume server evacuate from rack 2022-07-12 11:33:08 +05:00
Konstantin Lebedev
4236c36599 volume server evacuate to target server 2022-07-11 16:58:15 +05:00
chrislu
57975b93ff adjust used size reporting 2022-07-04 13:47:42 -07:00
chrislu
a79f5bd0d5 adjust used size reporting 2022-07-04 13:46:32 -07:00
guol-fnst
5bc895673e add getting bucket quota 2022-07-04 15:36:44 +08:00
guol-fnst
dd4664e610 fix usage 2022-07-04 11:31:39 +08:00
chrislu
141f662734 edge case checking when volume server does not have capacity to balance 
fix https://github.com/chrislusf/seaweedfs/issues/3257
 2022-07-01 12:16:18 -07:00
chrislu
bee3a7c798 fix tests 2022-07-01 12:09:42 -07:00
Konstantin Lebedev
5ed8165161 fix logic 
add option targetServer
https://github.com/chrislusf/seaweedfs/issues/3255
 2022-06-30 17:58:24 +05:00
Konstantin Lebedev
bcbdc4cb37 use const multipart uploads folder 
avoid error bucket NotEmpty if multipart uploads folder exist
 2022-06-29 16:21:16 +05:00
chrislu
1d0c53ea56 remote storage: stop supporting hdfs as a remote storage 2022-06-20 14:15:59 -07:00
石昌林
9e036df356 remove go.uber.org/atomic 2022-06-20 12:35:29 +08:00
石昌林
3dd60529c5 some code optimizations 2022-06-17 19:07:39 +08:00
石昌林
37df209195 add some unit tests and some code optimizes 2022-06-17 17:11:18 +08:00
石昌林
78b3728169 add s3 circuit breaker support for 'simultaneous request count' and 'simultaneous request bytes' limitations 
configure s3 circuit breaker by 'command_s3_circuitbreaker.go':
usage eg:
# Configure the number of simultaneous global (current s3api node) requests
s3.circuit.breaker -global -type count -actions Write -values 1000 -apply

# Configure the number of simultaneous requests for bucket x read and write
s3.circuit.breaker -buckets -type count -actions Read,Write -values 1000 -apply

# Configure the total bytes of simultaneous requests for bucket write
s3.circuit.breaker -buckets -type bytes -actions Write -values 100MiB -apply

# Disable circuit breaker config of bucket 'x'
s3.circuit.breaker -buckets x -enable false -apply

# Delete circuit breaker config of bucket 'x'
s3.circuit.breaker -buckets x -delete -apply
 2022-06-15 21:07:55 +08:00
chrislu
d12f431d98 collect volume server status 2022-06-12 11:56:23 -07:00
shichanglin5
a63559ee6c fix: fix command prompt information 2022-06-10 12:46:10 +08:00
chrislu
4fd5f96598 filer: remove replication, collection, disk_type info from entry metadata 
these metadata can change and are not used
 2022-06-06 00:39:35 -07:00
chrislu
f97acdd489 volume.fix.replication fix retry logic 
fix https://github.com/chrislusf/seaweedfs/issues/3136
 2022-06-03 08:45:29 -07:00
chrislu
6793bc853c help message when in simulation mode 2022-05-31 14:48:46 -07:00
chrislu
89948a373b fix error reporting for "Need to a hdd disk type" 
related to https://github.com/chrislusf/seaweedfs/issues/3128
 2022-05-31 12:43:55 -07:00
chrislu
5b9347c938 typo 2022-05-30 12:37:41 -07:00
chrislu
ca16fbf0ef shell: cluster.ps display filer group 2022-05-29 16:37:14 -07:00
Konstantin Lebedev
44f53ceda6 fix collectionIsMismatch charset 2022-05-16 13:23:23 +05:00
Konstantin Lebedev
10d435f2c2 fix skip loop 2022-05-16 13:16:27 +05:00
Konstantin Lebedev
279053572c avoid delete volume replica if collection mismatch 2022-05-16 13:07:05 +05:00
chrislu
271b5aed96 shell: volume.fsck add a note for -reallyDeleteFromVolume option 2022-05-15 11:07:04 -07:00
chrislu
94635e9b5c filer: add filer group 2022-05-01 21:59:16 -07:00
guosj
1a5de36257 fix return value 2022-04-29 11:19:36 +08:00
Chris Lu
b557faf175
Merge pull request #2967 from kmlebedev/vl_chk_disk_vlid 
volume.check.disk add param volumeId
 2022-04-26 22:09:06 -07:00
Konstantin Lebedev
d4343ab7da forcePurging desc 2022-04-25 23:11:56 +05:00
Konstantin Lebedev
ae56b2c00f change forcePurging to a pointer 2022-04-25 23:10:01 +05:00
Konstantin Lebedev
6d2fda27d2 delete missing data from volumes in one replica 2022-04-25 22:59:46 +05:00
Konstantin Lebedev
67476d830a volume.check.disk add param volumeId 2022-04-25 18:27:14 +05:00
Chris Lu
88a669dd19
Merge pull request #2858 from kmlebedev/fsck_replicas 
fsck replicas
 2022-04-19 00:17:43 -07:00
Konstantin Lebedev
2364fab927 volume.list show only readonly, collectionPattern and volumeId 2022-04-19 11:44:41 +05:00
Konstantin Lebedev
36c5a59ed8 add help 2022-04-18 19:36:14 +05:00
Konstantin Lebedev
1e35b4929f shell vacuum volume by collection and volume id 2022-04-18 18:40:58 +05:00
leyou240
89eb87c1d1
Merge branch 'master' into slices.SortFunc 2022-04-18 10:39:29 +08:00
justin
3551ca2fcf enhancement: replace sort.Slice with slices.SortFunc to reduce reflection 2022-04-18 10:35:43 +08:00
chrislu
460d56d283 shell: cluster.check prints out clock delta and network latency 2022-04-16 13:24:17 -07:00
chrislu
1f03fcccb1 fix nil in cluster_check shell command 
fix https://github.com/chrislusf/seaweedfs/issues/2905
 2022-04-12 08:47:27 -07:00
Konstantin Lebedev
76b1c5ce67 cleanupUploads use jwt token 2022-04-12 13:46:11 +05:00
Konstantin Lebedev
2dfbe210e0 avoid breaking loop in cleanupUploads if error is empty 2022-04-11 11:16:20 +05:00
Konstantin Lebedev
931cb9e581 use "cluster.raft.{ps,add,remove}" 2022-04-11 10:50:01 +05:00
Konstantin Lebedev
f5246b748d Merge branch 'new_master' into hashicorp_raft 
# Conflicts:
#	weed/pb/master_pb/master.pb.go
 2022-04-07 18:50:27 +05:00
Konstantin Lebedev
85d80fd36d fix removing old raft server 2022-04-07 15:31:37 +05:00
Konstantin Lebedev
357aa818fe add raft shell cmds 2022-04-06 15:23:53 +05:00
chrislu
bc888226fc erasure coding: tracking encoded/decoded volumes 
If an EC shard is created but not spread to other servers, the masterclient would think this shard is not located here.
 2022-04-05 19:03:02 -07:00
chrislu
4aae87f405 check missing hdd disk type 2022-04-04 14:48:00 -07:00
chrislu
6a2bcd03aa configure mount quota 2022-04-02 21:34:26 -07:00
chrislu
105578a2f2 skip pinging self for master and volume server 2022-04-01 20:25:35 -07:00
chrislu
4ecba915f3 add check between peers 2022-04-01 17:40:25 -07:00
chrislu
4b5c0e3fa9 check cluster connectivities 2022-04-01 17:27:49 -07:00
Konstantin Lebedev
7f1383a41e findExtraChunksInVolumeServers in consideration of replication 2022-04-01 14:45:41 +05:00
Konstantin Lebedev
3817e05dd0 fix collect filer files 2022-04-01 10:17:09 +05:00
Konstantin Lebedev
3cedb21bb7 skip new entities 2022-03-31 21:36:10 +05:00
Konstantin Lebedev
1d9e30d8c0 fsck replicas 2022-03-31 19:10:06 +05:00
Konstantin Lebedev
4f5b018a02 collecting only bucket volumes 2022-03-31 13:35:58 +05:00
chrislu
21e0898631 refactor: change masters from a slice to a map 2022-03-26 13:33:17 -07:00
banjiaojuhao
f7f2a597dd minor 2022-03-08 16:22:55 +08:00
chrislu
ede6ce44c6 fix test 2022-03-06 17:09:55 -08:00
chrislu
6e49e75a5b use logical number of files and sizes for statistics and quota 2022-03-04 18:47:44 -08:00
chrislu
95717d1006 simpler output 2022-02-24 13:50:08 -08:00
Chris Lu
6834df77a0
Merge pull request #2701 from guo-sj/fix_bugs_in_return_value 2022-02-23 19:23:37 -08:00
guosj
26f3ab8d4b fix a return bug in func (c *commandVacuum) Do 2022-02-24 09:04:38 +08:00
Chris Lu
38dcaaa76e
Merge pull request #2683 from guo-sj/fix_fsconfigure_bug 
fix minor bug in commandFsConfigure.Do
 2022-02-16 18:16:47 -08:00
guosj
ca121ecd66 fix minor bug in commandFsConfigure.Do 2022-02-17 09:57:52 +08:00
Chris Lu
64afbc5235
Merge pull request #2640 from Radtoo/fsck_with_delete 
Added basic deletion capabilities to fsck.
 2022-02-14 23:00:01 -08:00
Radtoo
724ce04b1c We picked the second flag's name. 2022-02-13 00:53:35 +01:00
Radtoo
fbe7ed7927 Revert the the flag to "reallyDeleteFromVolume". 2022-02-10 20:01:48 +01:00
chrislu
21aaa4c1f1 ec.encode: calculate free ec slots based on (maxVolumeCount-volumeCount) 
fix https://github.com/chrislusf/seaweedfs/issues/2642
 2022-02-08 01:51:13 -08:00
chrislu
13f6ec1c4e test checking ec distribution 2022-02-08 01:50:05 -08:00
chrislu
f34c2ff7c5 use embed txt 2022-02-08 01:14:27 -08:00
chrislu
c8c7c10c3f volume.tier.move: avoid double counting 
related to https://github.com/chrislusf/seaweedfs/issues/2637
 2022-02-08 00:57:35 -08:00
chrislu
f18803424a volume.balance: add delay during tight loop 
fix https://github.com/chrislusf/seaweedfs/issues/2637
 2022-02-08 00:53:55 -08:00
chrislu
85c1615b43 filer read empty file may cause OOM in some cases 
fix https://github.com/chrislusf/seaweedfs/issues/2641
 2022-02-07 23:08:54 -08:00
Radtoo
fbb14e0ea8 Adding separate toggle to purge absent vols 
While this toggle is basically required to clean out entries for deleted volumes, having a separate description + toggling this separately seems like a good idea so people get a chance to check if their volumes are all mounted/connected as expected.

Also renamed forcePurge to just purge.
 2022-02-06 23:46:52 +01:00
Radtoo
6b17f45da2 Also delete paths for which a volume is entirely absent. 2022-02-06 23:22:04 +01:00
Radtoo
fa0cfdfb7f Added basic deletion capabilities to fsck. 2022-02-06 17:05:58 +01:00
chrislu
6bee1e9714 [volume.check.disk] fix wrong logic to compare 
fix https://github.com/chrislusf/seaweedfs/issues/2626

minuend - subtrahend
 2022-02-01 07:48:28 -08:00
chrislu
b8490fe427 adjust volume count even when not applying the changes 2022-01-28 19:11:46 -08:00
divanikus
67e3fe996a async volumeTierMove 2022-01-26 18:22:31 +03:00
chrislu
9d0f58c329 skip printing fs configuration 2022-01-21 13:29:47 -08:00
chrislu
ce2049cdb6 refactoring, move genFn before saveFn 2022-01-21 12:08:58 -08:00
chrislu
e47f63d159 enforce bucket quota 2022-01-21 02:34:42 -08:00
chrislu
6e57d8d0de s3: check bucket usage and adjust read only according to quota 2022-01-21 02:15:27 -08:00
chrislu
606667f205 able to configure the quota for a bucket 2022-01-21 01:42:20 -08:00
chrislu
b1063162b6 display bucket quota 2022-01-21 00:55:04 -08:00
chrislu
f103491912 s3: list bucket size from weed shell 2022-01-21 00:26:49 -08:00
chrislu
ea57654e34 refactoring 2022-01-21 00:16:50 -08:00
chrislu
15c01d8b7f add some notes 2022-01-12 15:04:48 -08:00
chrislu
107a4884a8 shell: tighter memory allocation 2022-01-12 14:59:29 -08:00
chrislu
adfd54e7c4 fix compilation 2022-01-12 01:24:24 -08:00
chrislu
826a7b307e master: remove hard coded filer settings in master.toml 
fix https://github.com/chrislusf/seaweedfs/issues/2529
 2022-01-12 01:11:25 -08:00
chrislu
60dc450091 skip fixing read only volumes 
fix https://github.com/chrislusf/seaweedfs/issues/2562
 2022-01-06 09:52:28 -08:00
chrislu
9f9ef1340c use streaming mode for long poll grpc calls 
streaming mode would create separate grpc connections for each call.
this is to ensure the long poll connections are properly closed.
 2021-12-26 00:15:03 -08:00
banjiaojuhao
083bf3a137 filer server: add "datacenter, rack and datanode" for path specific configuration 2021-12-23 23:25:05 +08:00
chrislu
94e5c0a454 skip purging from ec volumes 2021-12-12 22:55:27 -08:00
chrislu
a2d3f89c7b add lock messages 2021-12-10 13:24:38 -08:00
chrislu
991a3dca0d rename file 2021-12-10 13:11:00 -08:00
chrislu
7f0a97c7b6 shell: volume.list supports different verbosity level 2021-12-05 21:54:40 -08:00
chrislu
e6c026db65 volume.fix.replication: fix misplaced volumes 
fix https://github.com/chrislusf/seaweedfs/issues/2416
 2021-12-05 16:56:25 -08:00
chrislu
53e2dee177 [volume.check.disk] was using the wrong source and target locations 
fix https://github.com/chrislusf/seaweedfs/issues/2268
 2021-12-05 14:32:04 -08:00
Chris Lu
f3a334965d Update command_remote_uncache.go 
minor
 2021-11-28 23:10:32 -08:00
Chris Lu
9ccfc1cfee Update command_volume_fsck.go 
minor
 2021-11-28 22:39:24 -08:00
Chris Lu
71a94267c9 Update command_volume_fsck.go 
remove unused code
 2021-11-28 22:37:11 -08:00
user
9668b15f38 Filtering by volume id is prioritized. 2021-11-11 18:18:56 +09:00
user
563a74a9eb Volume filter by collection pattern added. 2021-11-11 18:01:47 +09:00
user
dbb8003ce3 Volume filter function added. 2021-11-11 17:45:17 +09:00
user
c387fe957b Unused parameter removed. 2021-11-11 16:08:41 +09:00
Chris Lu
4729a57cc0 use constants 2021-11-08 17:47:56 -08:00
Chris Lu
a8b0f8864d fix help message 2021-11-07 12:38:35 -08:00
Chris Lu
5ea86ef1da Revert "master: rename grpc function KeepConnected() to SubscribeVolumeLocationUpdates()" 
This reverts commit af71ae11aa.
 2021-11-05 17:52:15 -07:00
Chris Lu
314c32514b add command to list filers 2021-11-05 17:39:41 -07:00
Chris Lu
00ae965d8d randomize a bit for ec shards distribution 2021-11-04 09:23:40 -07:00
Chris Lu
af71ae11aa master: rename grpc function KeepConnected() to SubscribeVolumeLocationUpdates() 2021-11-03 01:09:48 -07:00
Chris Lu
ab97b17e62 better printout 2021-11-02 23:45:47 -07:00
Chris Lu
5160eb08f7 shell: optionally read filer address from master 2021-11-02 23:38:45 -07:00
Chris Lu
18bfbf62fc add one unit test 2021-11-02 23:36:43 -07:00
Chris Lu
794375ca0a adjust help message since both fullPercent and quietFor are needed. 2021-11-01 17:22:47 -07:00
Chris Lu
24858507cc rename API to avoid confusion 2021-10-30 19:27:25 -07:00
Chris Lu
d774fa6c9a rename variable 2021-10-25 14:39:20 -07:00
Chris Lu
2539ba0b62 fix compilation 2021-10-25 14:38:48 -07:00
Chris Lu
5f2d7c1589 erasure coding: skip erasure coding if less than recommended 4 nodes 2021-10-25 14:38:11 -07:00
Chris Lu
5435027ff0 volume copy: stream out copying progress and avoid grpc request timeout 
fix https://github.com/chrislusf/seaweedfs/issues/2386
 2021-10-24 02:52:56 -07:00
Eng Zer Jun
a23bcbb7ec
refactor: move from io/ioutil to io and os package 
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
 2021-10-14 12:27:58 +08:00
Konstantin Lebedev
be4b3ed509 AclHandlers 2021-10-11 15:03:56 +05:00
Chris Lu
e862b2529a refactor 2021-10-01 12:10:11 -07:00
Konstantin Lebedev
5e64b22b45 check that the topology has been updated 2021-10-01 18:51:22 +05:00
Konstantin Lebedev
2cecde89c3 rename opt volumesPerStep 2021-10-01 00:17:54 +05:00
Konstantin Lebedev
fc51ffce2b https://github.com/chrislusf/seaweedfs/issues/1846 2021-09-30 20:24:24 +05:00
Chris Lu
ad5099e570 refactor 2021-09-19 12:02:23 -07:00
Chris Lu
2789d10342 go fmt 2021-09-14 10:37:06 -07:00
Chris Lu
119d5908dd shell: do not need to lock to see volume -h 2021-09-13 22:13:34 -07:00
Chris Lu
6cd1ce8b74 erasure coding: add cleanup step if anything goes wrong 2021-09-13 01:55:49 -07:00
Chris Lu
e5fc35ed0c change server address from string to a type 2021-09-12 22:47:52 -07:00
Chris Lu
0b4269b6a8 remember commands even if failed 2021-09-08 15:55:19 -07:00
Chris Lu
0128239c0f handle ipv6 addresses 2021-09-07 16:43:54 -07:00
Chris Lu
8e4b43a017 minor 2021-09-06 15:13:38 -07:00
Chris Lu
57a95887d2 remote.cache remote.uncache supports all mounted directories 2021-09-05 14:47:06 -07:00
Chris Lu
c735608685 obfuscate secret keys on display 2021-09-05 14:23:49 -07:00
Chris Lu
32e94de86a default auto trimming suffix to true 2021-09-05 13:27:35 -07:00
Chris Lu
60573fd3e2 option to map remote bucket to trimmed bucket name 2021-09-05 11:55:52 -07:00
Chris Lu
d57d4c5f8f shell: add remote.mount.buckets 2021-09-04 21:37:25 -07:00
Chris Lu
d983aa4c7d correct filtering 2021-09-04 13:58:14 -07:00
Chris Lu
49952ba905 fix cache/uncache filters 2021-09-04 13:45:39 -07:00
Chris Lu
df29281536 rename 2021-09-04 05:11:06 -07:00
Chris Lu
c3db389e42 refactor 2021-09-04 05:09:38 -07:00
Chris Lu
8707ef00a6 refactor 2021-09-04 05:02:20 -07:00
Chris Lu
63c84584bb refactor 2021-09-04 04:53:36 -07:00
Chris Lu
da49d25950 auto list of storage types 2021-09-04 00:18:21 -07:00
Chris Lu
38f73c8324 add gcs project id 2021-09-03 23:38:33 -07:00
Chris Lu
7ce97b59d8 go fmt 2021-09-01 02:45:42 -07:00
Chris Lu
3bd48c4f29 filer.remote.sync: exit when directory is unmounted 
this will not propagate the deletions back to the cloud
 2021-09-01 01:29:22 -07:00
Chris Lu
cca588429b shell: fs.meta.save skip saving system logs 2021-08-31 23:05:56 -07:00
Chris Lu
0ee208c1a4 adjust example 2021-08-31 02:04:32 -07:00
Chris Lu
97b7948cd9 adjust error message 2021-08-31 01:07:34 -07:00
Chris Lu
c9205458a1 cloud drive: create mount directory if not exists 2021-08-31 01:07:19 -07:00
Chris Lu
1dfcdc53b7 cloud drive: add storj, filebase 2021-08-30 17:43:14 -07:00
Chris Lu
701cc133f3 cloud drive: s3 add option for v4 signature 2021-08-30 17:28:33 -07:00
Chris Lu
f117f7bf54 adjust comment 2021-08-30 15:56:14 -07:00
Chris Lu
44d0ac95f6 add cluster id if already set 2021-08-29 22:19:46 -07:00
Chris Lu
18bf556d93 fix comments 2021-08-29 22:19:25 -07:00
Chris Lu
21f37a0a75 remove unused code 2021-08-29 22:19:11 -07:00
Chris Lu
001a472057 cloud mount: remote storage support hdfs 2021-08-29 18:41:29 -07:00
Chris Lu
975fbc9ee3 add wasabi example 2021-08-28 22:49:38 -07:00
Chris Lu
65a29ad42a add warning on unmount a folder 2021-08-28 02:25:48 -07:00
Chris Lu
a22f37b01c remove tencent region 2021-08-28 02:23:03 -07:00
Chris Lu
0d8936f6cd add aliyun example 2021-08-28 01:09:48 -07:00
Chris Lu
49a8dfb976 adjust default concurrent level 2021-08-26 17:05:56 -07:00
Chris Lu
6a0bb7106b cloud drive: parallelize remote storage downloading 2021-08-26 16:16:26 -07:00
Chris Lu
05a648bb96 refactor: separating out remote.proto 2021-08-26 15:18:34 -07:00
Chris Lu
c08ac536ed cloud drive: add support for Wasabi 
* disable md5, sha256 checking to avoid reading one chunk twice
* single threaded upload to avoid chunk swapping (to be enhanced later)
 2021-08-25 17:34:29 -07:00
Chris Lu
a7a914f120 cloud drive: add support for Baidu BOS 2021-08-24 23:46:33 -07:00
Chris Lu
2d805beed4 add help mesages 2021-08-24 23:28:35 -07:00
Chris Lu
c6c97bd83d add default env variables 2021-08-24 23:25:36 -07:00
Chris Lu
a19c728034 cloud drive: add support for Tencent COS 2021-08-24 23:19:45 -07:00
Chris Lu
19a81d25af cloud drive: add support for Aliyun OSS 2021-08-24 23:14:24 -07:00
Chris Lu
47d775cf68 cloud drive: add support for BackBlaze 2021-08-24 22:30:06 -07:00
Chris Lu
6bab20d862 cat entry content 2021-08-24 02:38:32 -07:00
Chris Lu
e9ebe24f2e cloud drive: add support for Azure 2021-08-24 01:18:30 -07:00
Chris Lu
00c4e06caa cloud drive: s3 configurable force path style 2021-08-23 03:30:41 -07:00
Chris Lu
2836a58d87 cloud drive: S3 supports storage class 2021-08-23 02:18:59 -07:00
Chris Lu
ca35a77e85 adjust help message 2021-08-23 00:37:55 -07:00
Chris Lu
258063de26 cloud drive: add google cloud storage 2021-08-23 00:29:27 -07:00
Chris Lu
2158d4fe4d adjust help message 2021-08-21 02:17:10 -07:00
Chris Lu
ccb3df41f8 elide secret key 2021-08-17 11:27:08 -07:00
Chris Lu
49b5e47bd1 retry forever with filer.remote.sync, and some refactoring 2021-08-15 12:38:26 -07:00
Chris Lu
a539d64896 refactor 2021-08-15 12:09:54 -07:00
Chris Lu
ec989b0377 remove println 2021-08-15 02:10:27 -07:00
Chris Lu
9462f5129a shell: add "remote.meta.sync" 2021-08-15 01:53:46 -07:00
Chris Lu
c34747c79d rename, fix wrong logic. 2021-08-14 21:46:34 -07:00
Chris Lu
cb53802752 adjust help message 2021-08-14 15:55:53 -07:00
Chris Lu
889b143fa7 adjust modification detection logic 2021-08-14 15:44:47 -07:00
Chris Lu
708debca14 remote.cache and uncache: more flexible options to select files to cache or uncache 2021-08-14 15:11:55 -07:00
Chris Lu
0db2517994 go fmt 2021-08-14 02:55:44 -07:00
Chris Lu
0f7d4556d8 shell: volume.tier.move makes up changes if volume move failed 2021-08-13 03:09:28 -07:00
Chris Lu
333cdce485 add verbose message 2021-08-13 02:57:14 -07:00
Chris Lu
057ef429ac format 2021-08-10 12:33:29 -07:00
Chris Lu
b63b042afc dedup keeps the largest replica 2021-08-10 12:30:41 -07:00
Chris Lu
0526db12e2 do not treat read only volumes differently 2021-08-10 11:37:12 -07:00
Chris Lu
e50a5b8e28 minor: print disk type 2021-08-10 11:10:09 -07:00
Chris Lu
db6275a0c8 print out balance ratio 2021-08-10 10:43:42 -07:00
Chris Lu
85832d02c0 wait for goroutines 2021-08-10 04:13:12 -07:00
Chris Lu
8ff6c9a0c6 output format 2021-08-10 03:25:18 -07:00
Chris Lu
48f448ee09 parallelize tier move 2021-08-10 03:08:29 -07:00