Commit Graph

8634 Commits

Author SHA1 Message Date
chrislu
c839ce1b19 s3 sink use s3 upload manager
fix https://github.com/seaweedfs/seaweedfs/issues/3531
2022-08-26 23:47:12 -07:00
chrislu
3f3a1341d8 make CodeQL happy 2022-08-26 17:09:11 -07:00
chrislu
301b49b63f atomic operation 2022-08-26 17:04:11 -07:00
chrislu
57a46f46a0 nano level precision 2022-08-26 16:55:15 -07:00
chrislu
8dae81c5ed Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-08-26 16:49:51 -07:00
chrislu
e0f4366f4c more detailed error
related to https://github.com/seaweedfs/seaweedfs/issues/3528
2022-08-26 16:49:48 -07:00
Patrick Schmidt
5df105b1f9
Fix a few data races when reading files in mount (#3527) 2022-08-26 16:41:37 -07:00
chrislu
f5156cf3a8 rename varaible 2022-08-26 10:23:42 -07:00
Konstantin Lebedev
e90ab4ac60
avoid race conditions for OnPeerUpdate (#3525)
https://github.com/seaweedfs/seaweedfs/issues/3524
2022-08-26 10:18:49 -07:00
Konstantin Lebedev
4f7a1f67cd
avoid race conditions for diskLocation.MaxVolumeCount (#3526) 2022-08-26 08:41:42 -07:00
Konstantin Lebedev
913a5d0dd4
add target dev_race to MakeFile (#3522) 2022-08-26 01:28:03 -07:00
chrislu
6f483a4d36 atomic add
fix https://github.com/seaweedfs/seaweedfs/issues/3514
2022-08-25 22:20:34 -07:00
chrislu
7394f7feee minor: adjust ordering 2022-08-25 00:20:23 -07:00
chrislu
dbf0de4ce1 minor clean up 2022-08-25 00:19:08 -07:00
Patrick Schmidt
2930263dfd
Fix race conditions during in-flight size checks (#3505) 2022-08-24 20:03:34 -07:00
Patrick Schmidt
7b424a54dc
Add raft server access mutex to avoid races (#3503) 2022-08-24 09:49:05 -07:00
Chris Lu
f7e4359b59
Temporarily enable cgo
Temporarily enable cgo following findings from https://github.com/seaweedfs/seaweedfs/discussions/3152 by https://github.com/dyus

Resolve build issue:
```
# github.com/mattn/go-ieproxy
/go/pkg/mod/github.com/mattn/go-ieproxy@v0.0.3/ieproxy.go:36:9: undefined: getConf
/go/pkg/mod/github.com/mattn/go-ieproxy@v0.0.3/ieproxy.go:41:9: undefined: reloadConf
/go/pkg/mod/github.com/mattn/go-ieproxy@v0.0.3/ieproxy.go:48:2: undefined: overrideEnvWithStaticProxy
/go/pkg/mod/github.com/mattn/go-ieproxy@v0.0.3/ieproxy.go:53:13: psc.findProxyForURL undefined (type *ProxyScriptConf has no field or method findProxyForURL, but does have FindProxyForURL)
```
2022-08-24 02:24:21 -07:00
Konstantin Lebedev
f7aeb06544
s3: report metadata if the directory is explicitly created (#3498)
* replace mkdir to mkFile

* ContentLength must be zero

* revert mkDir

* Seaweedfs-Is-Directory-Key return metadata
2022-08-24 00:15:44 -07:00
Guo Lei
c57c79a0ab
optimiz commitig compact (#3388)
* optimiz vacuuming volume

* fix bugx

* rename parameters

* fix conflict

* change copyDataBasedOnIndexFile to an instance method

* close needlemap

* optimiz commiting Vacuum volume for  leveldb index

* fix bugs

* fix leveldb loading bugs

* refactor

* fix leveldb loading bug

* add leveldb recovery

* add test case for levelDB

* modify test case to cover all the new branches

* use one tmpNm instead of two instances

* refactor

* refactor

* move setWatermark to the end

* add test for watermark and updating leveldb

* fix error logic

* refactor, add test

* check nil before close needlemapeer
add test case
fix metric bug

* add tests, fix bugs

* adjust log level
remove wrong test case
refactor

* avoid duplicate  updating metric for leveldb index
2022-08-23 23:53:35 -07:00
chrislu
10414fd81c ping timeout at 15 seconds
this 72 minute timeout setting seems unreasonably long

15 seconds is around the time when a new raft leader should be elected.
2022-08-23 23:28:16 -07:00
askeipx
2e78a522ab
remove old raft servers if they don't answer to pings for too long (#3398)
* remove old raft servers if they don't answer to pings for too long

add ping durations as options

rename ping fields

fix some todos

get masters through masterclient

raft remove server from leader

use raft servers to ping them

CheckMastersAlive for hashicorp raft only

* prepare blocking ping

* pass waitForReady as param

* pass waitForReady through all functions

* waitForReady works

* refactor

* remove unneeded params

* rollback unneeded changes

* fix
2022-08-23 23:18:21 -07:00
Guo Lei
762dc219a6
fix deletecount bug (#3499) 2022-08-23 08:53:24 -07:00
chrislu
21a933c932 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-08-23 01:52:32 -07:00
chrislu
f0b4a7659a fix test 2022-08-23 01:52:29 -07:00
Patrick Schmidt
cda8cc22bc
Implement lseek syscall in FUSE (#3491)
See the man page of lseek:
https://man7.org/linux/man-pages/man2/lseek.2.html
2022-08-23 01:47:18 -07:00
chrislu
bf88006037 format 2022-08-23 01:20:45 -07:00
chrislu
9fce75607d s3: report http.StatusOK if the directory is explicitly created
fix https://github.com/seaweedfs/seaweedfs/issues/3457
2022-08-23 01:16:46 -07:00
famosss
911475526c
fix: TestCommandEcBalanceSmall Unit test fails when CommandEnv is nil (#3497) 2022-08-22 23:54:51 -07:00
famosss
7eb15b1969
fix:Handle preflight cors requests (#3496) 2022-08-22 23:39:34 -07:00
chrislu
676e27c589 shell: stop long running jobs if lock is lost 2022-08-22 14:12:23 -07:00
chrislu
601ba5fb68 master: disable locking if not leader 2022-08-22 14:11:48 -07:00
chrislu
57e7582c36 refactoring 2022-08-22 14:11:13 -07:00
chrislu
7ea2c24c5d metadata processing use retry instead of retryForEver 2022-08-22 11:25:40 -07:00
dependabot[bot]
741a013689
Bump github.com/Shopify/sarama from 1.35.0 to 1.36.0 (#3487) 2022-08-22 08:44:56 -07:00
dependabot[bot]
ebc1b478b1
Bump wangyoucao577/go-release-action from 1.22 to 1.30 (#3484) 2022-08-22 08:36:15 -07:00
dependabot[bot]
66a1d6b50f
Bump actions/dependency-review-action from 2.0.4 to 2.1.0 (#3485) 2022-08-22 08:35:59 -07:00
dependabot[bot]
6ec81f09f7
Bump github.com/tidwall/gjson from 1.14.2 to 1.14.3 (#3486) 2022-08-22 08:35:36 -07:00
dependabot[bot]
622521c247
Bump github.com/ydb-platform/ydb-go-sdk/v3 from 3.33.0 to 3.35.1 (#3488) 2022-08-22 08:35:06 -07:00
dependabot[bot]
6d4846a29a
Bump github.com/fclairamb/ftpserverlib from 0.19.0 to 0.19.1 (#3489) 2022-08-22 08:34:43 -07:00
dependabot[bot]
d9991aeaef
Bump modernc.org/sqlite from 1.18.0 to 1.18.1 (#3490) 2022-08-22 08:34:28 -07:00
dependabot[bot]
9076a0051c
Bump actions/checkout from 3 to 3.0.2 (#3483) 2022-08-22 08:33:53 -07:00
Patrick Schmidt
f875031f06
Reuse readDataByFileHandle in Read call (#3482) 2022-08-22 08:24:06 -07:00
famosss
bdba3da2e4
Handle preflight cors requests (#3481) 2022-08-22 08:21:38 -07:00
chrislu
f4b52d4c52 fix format 2022-08-22 00:20:21 -07:00
chrislu
b4131a3a72 filer.remote.sync: parallelize uploading to remove object storage 2022-08-22 00:01:48 -07:00
chrislu
f6b92b8561 refactor 2022-08-22 00:00:51 -07:00
chrislu
3c8e95eeb3 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-08-21 23:26:09 -07:00
chrislu
8bac903d29 mount: add locking to prevent nil pointer
fix https://github.com/seaweedfs/seaweedfs/issues/3479
2022-08-21 23:26:06 -07:00
qzh
74b53729e1
feat(weed.move): add a speed limit parameter of moving files (#3478)
* feat(weed.move): add a speed limit parameter of moving files

* fix(weed.move): set the default value of ioBytePerSecond to vs.compactionBytePerSecond

Co-authored-by: zhihao.qu <zhihao.qu@ly.com>
2022-08-21 23:08:31 -07:00
chrislu
7807f6641a minor 2022-08-21 22:54:56 -07:00