mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-12-11 15:59:06 +08:00
Code Issues Packages Projects Releases Wiki Activity
9,901 Commits 32 Branches 296 Tags 150 MiB
c50b6387f6
Commit Graph

378 Commits

Author SHA1 Message Date
Chris Lu
4e31ca9507 upgrade junit to 4.13.1 2020-10-12 11:37:15 -07:00
dependabot[bot]
b57d435abe
Bump junit from 3.8.1 to 4.13.1 in /other/java/s3copier 
Bumps [junit](https://github.com/junit-team/junit4) from 3.8.1 to 4.13.1.
- [Release notes](https://github.com/junit-team/junit4/releases)
- [Changelog](https://github.com/junit-team/junit4/blob/main/doc/ReleaseNotes4.13.1.md)
- [Commits](https://github.com/junit-team/junit4/commits/r4.13.1)

Signed-off-by: dependabot[bot] <support@github.com>
 2020-10-12 18:14:12 +00:00
Chris Lu
253323ce6d Hadoop 1.5.0 2020-10-12 11:13:38 -07:00
hilimd
2ff727a32d
Merge pull request #25 from chrislusf/master 
sync
 2020-10-12 11:01:13 +08:00
Chris Lu
d5fcb0f474 Hadoop: printout the entry creation error message 2020-10-11 15:11:16 -07:00
Chris Lu
ea13e0523b fix logging 2020-10-11 14:59:48 -07:00
Chris Lu
8e2f3b510e Hadoop client: 1.4.9 2020-10-10 15:43:04 -07:00
hilimd
b9a446839a
Merge pull request #24 from chrislusf/master 
sync
 2020-10-10 16:20:26 +08:00
Chris Lu
912ef2bc53 Hadoop: remove unused variable bufferSize 2020-10-10 01:14:17 -07:00
Chris Lu
9449100757 adjust throwing exception 2020-10-10 01:06:54 -07:00
Chris Lu
d3d3609a03 adjust logging 2020-10-10 01:04:31 -07:00
Chris Lu
6fc272f913 adjust throwing exception 2020-10-10 01:03:04 -07:00
Chris Lu
00a75d7c99 Hadoop: fix reading file tail 2020-10-10 00:57:57 -07:00
Chris Lu
e1a3ffcdbf Hadoop: add exponential back off for failed reads 2020-10-10 00:57:57 -07:00
hilimd
411e49f964
Merge pull request #23 from chrislusf/master 
sync
 2020-10-10 13:07:57 +08:00
limd
ac162fc857 hdfs: Hadoop on SeaweedFS: create empty file 2020-10-10 13:07:27 +08:00
Chris Lu
a34bad2cee moving grafana dashboard here 2020-09-30 13:30:21 -07:00
Chris Lu
436ab7d811 HCFS: 1.4.8 2020-09-27 23:20:08 -07:00
Chris Lu
4929d0634e Hadoop on SeaweedFS: create empty file 
fix https://github.com/chrislusf/seaweedfs/issues/1494
 2020-09-25 10:57:43 -07:00
limd
95bfec4931 hadoop: filesystem cannot create file 
issues: https://github.com/chrislusf/seaweedfs/issues/1494
 2020-09-26 01:38:48 +08:00
Chris Lu
1012df7bb5 switch hardlink id from int64 to bytes 2020-09-24 11:11:42 -07:00
Chris Lu
5e239afdfc hardlink works now 2020-09-24 03:06:48 -07:00
Chris Lu
cb427d48fa filer report metrics configuration 2020-09-17 06:46:51 -07:00
Chris Lu
5eee4983f3 1.4.7 hdfs configurable fs.seaweed.buffer.size 2020-09-16 17:18:18 -07:00
Chris Lu
387ab6796f filer: cross cluster synchronization 2020-09-09 11:21:23 -07:00
Chris Lu
bba90ff3c8 read filer signature 2020-09-05 22:52:15 -07:00
Chris Lu
ca658a97c5 add signatures to messages to avoid double processing 2020-08-28 23:48:48 -07:00
Chris Lu
48ed7f8a32 generated files 2020-08-16 21:25:24 -07:00
Chris Lu
f5d6b8b777 Hadoop HCFS 1.4.6 random access file 2020-08-16 21:17:03 -07:00
Chris Lu
13bfe5deef same logic for reading random access files from Go 2020-08-16 21:14:39 -07:00
Chris Lu
20e0bae5d1 add for testing 2020-08-16 09:03:18 -07:00
Chris Lu
24bfd26719 make it easy to test randome access file 2020-08-15 14:37:07 -07:00
Chris Lu
4f195a54ca 1.4.5 
change the scope for hadoop depencies to provided
 2020-08-09 16:34:47 -07:00
Chris Lu
15dc0a704d Revert "add read ahead input stream" 
This reverts commit b3089dcc8e.
 2020-08-03 09:11:24 -07:00
Chris Lu
b3089dcc8e add read ahead input stream 2020-08-03 09:06:09 -07:00
Chris Lu
53190a9972 fix compilation 2020-08-03 09:05:49 -07:00
Chris Lu
d6073f6386 ensure GC 2020-08-03 00:40:23 -07:00
Chris Lu
9981748498 only try to cache chunk manifest chunks 2020-08-02 23:50:44 -07:00
Chris Lu
0ed1f43d29 decompress after decrypt if necessary 
skip any decompress error
 2020-08-02 23:50:23 -07:00
Chris Lu
e1f070a9a1 Hadoop: 1.4.4 2020-08-01 12:42:41 -07:00
Chris Lu
4d9da157bc HDFS: read gzip content 2020-08-01 11:36:29 -07:00
Chris Lu
8cc35e2c13 fix wrong decompression logic 2020-08-01 11:19:57 -07:00
Chris Lu
6b41c5250b Hadoop file system: 1.4.3 
added buffered fs input stream
 2020-07-29 21:33:10 -07:00
Chris Lu
703057bff9 mirror changes from hdfs2 2020-07-29 20:15:43 -07:00
Chris Lu
a6d0f962e8 add buffered seaweed input stream 2020-07-29 20:10:04 -07:00
Chris Lu
5080bc1d69 debug 2020-07-29 18:26:26 -07:00
Chris Lu
daf8c0c8ce cache vid locations 2020-07-29 18:25:52 -07:00
Chris Lu
b684c312d2 minor 2020-07-29 18:25:35 -07:00
Chris Lu
735912e0f5 enable read cache 2020-07-29 17:39:44 -07:00
Chris Lu
6f058b3791 HCFS: 1.4.1 2020-07-22 22:56:53 -07:00
Chris Lu
648ef566c4 HCFS: avoid lock bottleneck 2020-07-22 22:54:06 -07:00
Chris Lu
6839f96c0c simplify 2020-07-22 22:52:49 -07:00
Chris Lu
54f9f9138e HCFS: 1.4.0 2020-07-20 23:25:08 -07:00
Chris Lu
64926d2345 disable ChunkCache 2020-07-20 21:18:58 -07:00
Chris Lu
ae3e6d8244 remove changing buffer size 2020-07-20 20:27:19 -07:00
Chris Lu
b380b9e5d7 shared http client 2020-07-20 18:27:11 -07:00
Chris Lu
0e341a2a9a error logging 2020-07-20 18:26:59 -07:00
Chris Lu
cacc601cc8 ensure changing buffer size requirements 2020-07-20 18:26:48 -07:00
Chris Lu
aee6d89350 before writing and after reading file chunks 2020-07-20 18:26:17 -07:00
Chris Lu
6debe3c3ec use 1000 as merge factor 2020-07-20 18:25:26 -07:00
Chris Lu
1d724ab237 hdfs: support read write chunk manifest 2020-07-20 03:29:17 -07:00
Chris Lu
f90d2c93c9 1.3.9 remove logs 2020-07-17 16:11:57 -07:00
Chris Lu
3abd74b1d7 1.3.8 2020-07-17 15:46:20 -07:00
Chris Lu
0b9d73fb7c async copy 2020-07-17 15:46:20 -07:00
Chris Lu
54445b207d Hadoop: 1.3.6 2020-07-16 16:20:01 -07:00
Chris Lu
d6d95765f9 reduce chunk cache size 2020-07-16 16:19:01 -07:00
Chris Lu
2629da2cb9 simplify inputstream 2020-07-16 10:54:29 -07:00
Chris Lu
c9b50e8a22 copy changes to hadoop2 2020-07-16 09:05:34 -07:00
李明达
74456b3d5e 1、Add SeaweedFS implementation of Hadoop AbstractFileSystem. The implementation delegates to the existing SeaweedFS FileSystem and is only necessary for use with Hadoop 2.x/3.x. Configuration example in Hadoop core-site.xml file: 
<property>
    <name>fs.AbstractFileSystem.seaweedfs.impl</name>
    <value>seaweed.hdfs.SeaweedAbstractFileSystem</value>
 </property>

2、Fix hiveserver2 startup NullPointerException
 2020-07-16 16:47:09 +08:00
Chris Lu
64d7d5fe89 increase inbound message size 2020-07-16 00:33:54 -07:00
Chris Lu
7bca72deed reuse bytebuffer 2020-07-15 23:33:31 -07:00
Chris Lu
2286d27730 rename 2020-07-15 14:19:15 -07:00
Chris Lu
86c8137546 1.3.4 2020-07-15 13:49:06 -07:00
Chris Lu
22f8aae028 fix compilation 2020-07-15 13:27:10 -07:00
Chris Lu
bc3be0bb37 Hadoop: 1.3.3 
improve memory efficiency
 2020-07-15 13:25:44 -07:00
Chris Lu
316e853e0e option to disable chunk cache 2020-07-15 09:51:51 -07:00
Chris Lu
0db4204c81 add is_file_chunks to filer.proto 2020-07-14 22:42:52 -07:00
Chris Lu
fc9af962ec 1.3.2 2020-07-12 14:51:24 -07:00
Chris Lu
4bd8f3281e 1.3.0 
* add option to include last entry
*
 2020-07-11 22:24:02 -07:00
Chris Lu
578f316173 refactoring 2020-07-11 15:33:31 -07:00
Chris Lu
70d8a3a1d3 add SubscribeLocalMetadata without checking persisted meta logs 2020-07-05 15:50:07 -07:00
Chris Lu
37d5b3ba12 replication: pass isFromOtherCluster also to EventNotification 
EventNotification is consistent with message queue and metadata logs.
 2020-07-01 08:06:20 -07:00
Chris Lu
31e23e9783 filer: support active<=>active filer replication 2020-06-30 22:53:57 -07:00
Chris Lu
97239ce6f1 rename filechunk is_gzipped to is_compressed 2020-06-20 08:15:49 -07:00
Chris Lu
ca3516ac6d adjust protoc 2020-06-20 08:00:25 -07:00
Chris Lu
ce79ec0bd4 HCFS: 1.2.9 fix concurrent modification 2020-06-18 07:52:32 -07:00
Chris Lu
222f93e816 possibly fix concurrent access to entry object in SeaweedOutputStream 2020-06-17 14:59:43 -07:00
Chris Lu
8dfaaeabfd HCFS: 1.2.8 fix hbase related bugs 
1. SeaweedFileSystem.listStatus need to work with file also
2. SeaweedRead readChunkView has wrong len
 2020-05-24 17:07:34 -07:00
Chris Lu
dfccc3c263 able to read chan and write chan 2020-05-08 02:47:22 -07:00
Chris Lu
1e3e4b3072 add broker connects to filer 2020-05-05 02:05:28 -07:00
Chris Lu
972f437225 HCFS 1.2.7 2020-04-26 05:44:56 -07:00
Chris Lu
b52b8ec685 Hadoop: fix entry not found for HCFS 
also fix cipher related changes.
 2020-04-26 05:21:54 -07:00
Chris Lu
e24b25de78 async meta caching: can stream updates now 2020-04-21 21:16:13 -07:00
Chris Lu
bda82f61bc filer: able to append to a file 2020-04-17 02:28:09 -07:00
Chris Lu
7764e0465c refactoring 2020-04-12 21:00:55 -07:00
Chris Lu
6f948e4887 remove configurable topics folder location 2020-04-12 13:07:59 -07:00
Chris Lu
ec2eb8bc48 add If-None-Match and If-Modified-Since 
fix https://github.com/chrislusf/seaweedfs/issues/1269
 2020-04-08 08:12:00 -07:00
Chris Lu
2d43f85577 watch entries with common path prefix 2020-04-05 12:51:21 -07:00
Chris Lu
bf270d9e8c filer: able to tail meta data changes 2020-04-05 00:51:16 -07:00
Chris Lu
50a5018b7f writing meta logs is working 2020-03-30 01:19:33 -07:00
Chris Lu
5ebf232e79 HDFS: 1.2.6 2020-03-29 01:40:04 -07:00
Chris Lu
b656e05aaf HDFS: support chunk cache 2020-03-29 00:55:40 -07:00
Chris Lu
ae2309dc58 Update SeaweedRead.java 2020-03-28 23:58:59 -07:00
Chris Lu
3d3bab2447 add notes 2020-03-28 23:43:24 -07:00
Chris Lu
c4bea45099 S3 API: fix DeleteMultipleObjectsHandler 
fix https://github.com/chrislusf/seaweedfs/issues/1241
 2020-03-20 14:17:31 -07:00
Chris Lu
de1ba85346 HDFS support encrypted data storage 2020-03-14 00:27:57 -07:00
Chris Lu
2e3f6ad3a9 filer: remember content is gzipped or not 2020-03-08 21:39:33 -07:00
Chris Lu
13e215ee5c filer: option to encrypt data on volume server 2020-03-06 00:49:47 -08:00
Chris Lu
555413d9fc weed queue starts 2020-02-27 00:07:13 -08:00
Chris Lu
0841bedb15 move filer assign volume grpc errror to response 2020-02-25 17:15:09 -08:00
Chris Lu
bc38b72a20 s3: implemented DeleteMultipleObjects 2020-02-25 14:38:36 -08:00
Chris Lu
6ab7368ef2 filer: dynamically create bucket under /buckets folder 2020-02-24 22:28:45 -08:00
Chris Lu
d8dec2323b s3: move buckets folder configuration to filer 2020-02-24 14:34:14 -08:00
Chris Lu
c48fc8b4de grpc send error via response instead of grpc error 2020-01-25 09:17:19 -08:00
Chris Lu
6a5c037099 fix http range requests 2020-01-22 22:59:12 -08:00
Chris Lu
d4e75a0d18 filer: option to create file only if it is new, O_EXCL 2020-01-22 11:42:40 -08:00
Chris Lu
69ffcbdcc0 HCFS: release 1.2.4 2019-12-14 11:04:36 -08:00
Chris Lu
987108a2b1 HCFS: streaming list files 2019-12-13 10:04:53 -08:00
Chris Lu
0fa1269bc7 filer: streaming file listing 2019-12-13 00:22:37 -08:00
Chris Lu
e9dd5a602a HCFS: 1.2.3 
shade org.apache.http in https://github.com/chrislusf/seaweedfs/issues/1080
 2019-10-11 23:56:04 -07:00
Chris Lu
7057e7a076 HCFS: 1.22 
fix put gives java.lang.NoSuchFieldError: INSTANCE related to Cloudera CDH Hadoop https://github.com/chrislusf/seaweedfs/issues/1080
 2019-10-11 21:44:44 -07:00
Chris Lu
15f968b054 HCFS: avoid possible jar problem 2019-10-09 18:09:08 -07:00
Chris Lu
a999ed94d0 update hdfs client 2019-10-08 22:49:10 -07:00
Chris Lu
ae53f63680 filer: recursive deletion optionally ignoring any errors 
fix https://github.com/chrislusf/seaweedfs/issues/1062
 2019-09-11 20:26:20 -07:00
Chris Lu
d361ecb23b HCFS: 1.1.6 2019-09-03 01:31:31 -07:00
Chris Lu
fd509c3844 HCFS: working with HBase 2019-09-03 00:50:28 -07:00
Chris Lu
60c9215a00 HCFS: avoid verbose file not found exception stack trace 2019-09-01 21:40:26 -07:00
Chris Lu
af8b413a9c HCFS: fix mkdirs NPE 2019-09-01 12:25:06 -07:00
Chris Lu
cb299dfaa2 HCFS: use latest grpc versions, separate hadoop2 and hadoop3 2019-09-01 03:46:51 -07:00
Chris Lu
170ee6ef0f tmp 2019-08-29 23:29:10 -07:00
Chris Lu
8da5d5b094 filer.copy: use filer settings, avoid unnecessary command line options 
fix https://github.com/chrislusf/seaweedfs/issues/968
 2019-06-23 01:57:35 -07:00
Chris Lu
82b0759493 filer: migrating filer store from persisting shorter structured file id instead of a string 2019-05-17 02:03:23 -07:00
Chris Lu
b3b42bc947 replicate need to include new entry path 2019-04-16 00:44:31 -07:00
Chris Lu
97406333a5 support atomic renaming for mysql/postgres filer store 2019-03-30 23:08:29 -07:00
Chris Lu
097b7a321e HDFS: move to 1.0.8 2019-02-20 00:57:31 -08:00
Chris Lu
6ed69de6bd HDFS: add jwt auth for uploading 2019-02-19 14:26:59 -08:00
Chris Lu
58d4088db4 HDFS: add tls secured grpc 2019-02-19 11:57:25 -08:00
Chris Lu
74fb237727 benchmark can work in secure mode 2019-02-15 00:09:19 -08:00
Chris Lu
9123d799b8 refactor 2018-12-27 23:40:23 -08:00
Chris Lu
319ab6d98f refactoring 2018-12-27 23:29:51 -08:00
Chris Lu
253f190f48 weed mount add symlink support 2018-12-25 22:45:44 -08:00
Chris Lu
9546104961 HCFS 1.05 2018-12-18 20:23:40 -08:00
Chris Lu
2c72447078 HCFS: bug fix for listEntries 
fix https://github.com/chrislusf/seaweedfs/issues/797
 2018-12-18 20:22:51 -08:00
Chris Lu
e716ec9908 HCFS 1.04 2018-12-17 20:40:10 -08:00
Chris Lu
cab7fc7738 fix directory creation 
fix https://github.com/chrislusf/seaweedfs/issues/794
 2018-12-17 20:37:55 -08:00
Chris Lu
d11cfdb356 paginate through large folders 2018-12-17 01:25:02 -08:00
Chris Lu
21315f709d HCFS 1.0.2 2018-12-14 09:16:21 -08:00
Chris Lu
36d1e739e6 1.11 2018-12-08 00:32:34 -08:00
Chris Lu
adf12c8825 refactoring 2018-12-07 02:23:45 -08:00
Chris Lu
1bfb96f34d optimization for reading whole chunk with gzip encoding 2018-12-07 01:57:55 -08:00
Chris Lu
6946c51430 avoid range request for gzipped content 2018-12-07 01:23:30 -08:00
Chris Lu
6b39effe7f add a local test 2018-12-07 01:23:02 -08:00
Chris Lu
09383860e9 HCFS recursively delete files 2018-12-04 22:57:23 -08:00
Chris Lu
060ce1d9a2 HCFS: adjust file modification time to milliseconds 2018-12-04 01:11:13 -08:00
Chris Lu
5f8c8caec6 HCFS: chmod 2018-12-04 01:10:25 -08:00
Chris Lu
d5197d6a50 HCFS support chown 2018-12-04 00:48:11 -08:00
Chris Lu
091f6297c4 add debug 2018-12-04 00:28:10 -08:00
Chris Lu
aea86cbf89 add @Override 2018-12-04 00:28:03 -08:00
Chris Lu
ce566f579e add juint 2018-12-03 22:12:31 -08:00
Chris Lu
c85ee7c0fd HCFS: read concatenated files 2018-12-03 22:12:20 -08:00
Chris Lu
4119c61df8 HCFS can read files 2018-12-03 20:25:57 -08:00
Chris Lu
5b50182658 put file is working 2018-12-03 01:37:29 -08:00
Chris Lu
d3be8e022f HCFS can ls, mkdir 2018-12-02 22:44:49 -08:00
Chris Lu
4263805c78 resolve java jar dependencies 2018-12-02 15:16:49 -08:00
Chris Lu
c6a567acab add SeaweedInputStream 2018-11-29 01:04:40 -08:00
Chris Lu
021f5d689b copy the visible chunks logic from Go implementation 2018-11-26 00:32:04 -08:00
Chris Lu
b089f1a492 SeaweedWrite close http client 2018-11-26 00:00:55 -08:00
Chris Lu
11eb014311 SeaweedFileSystem add SeaweedOutputStream to write 2018-11-25 23:49:05 -08:00
Chris Lu
f970abf14a WIP: adding SeaweedOutputStream for write 2018-11-25 18:01:57 -08:00
Chris Lu
6ecefad692 SeaweedFileSystem add rename 2018-11-25 14:48:17 -08:00
Chris Lu
1cbd53c01c WIP SeaweedFileSystem added mkdirs, getFileStatus, listStatus, delete 2018-11-25 13:43:26 -08:00
Chris Lu
d9871e92d2 adjust module name 2018-11-24 03:24:40 -08:00
Chris Lu
76cba561cf starting with hadoop compatible 2018-11-24 03:22:25 -08:00
Chris Lu
2697d6a4fb fix deleting bucket 2018-11-08 22:15:21 -08:00
Chris Lu
70008b093e test deletes 2018-11-08 22:08:11 -08:00
Chris Lu
a4ceb051a7 use MD5 for ETag to be consistent with Amazon S3 2018-11-08 21:41:02 -08:00
Chris Lu
f6d8525d1d working S3 multipart uploads 2018-09-12 00:46:12 -07:00
Chris Lu
73f331df84 add java example of copying to weed s3 end point 2018-09-02 14:19:24 -07:00