Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Removed manual accounting of field's count #12842

Open
wants to merge 58 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
58 commits
Select commit Hold shift + click to select a range
c7ff301
Removed manual accounting of field's count and added option to change…
igeni Mar 23, 2024
47e817c
Merge branch 'master' into remove-manual-accounting
glenn-jocher Apr 9, 2024
5851298
Merge branch 'master' into remove-manual-accounting
UltralyticsAssistant Apr 14, 2024
bdad2c5
Merge branch 'master' into remove-manual-accounting
UltralyticsAssistant Apr 18, 2024
b755550
Merge branch 'master' into remove-manual-accounting
UltralyticsAssistant Apr 27, 2024
ce3258d
Merge branch 'master' into remove-manual-accounting
UltralyticsAssistant Apr 28, 2024
70dae61
Auto-format by https://ultralytics.com/actions
UltralyticsAssistant Apr 28, 2024
a6f5c8c
Merge branch 'master' into remove-manual-accounting
UltralyticsAssistant May 5, 2024
623c7cd
Merge branch 'master' into remove-manual-accounting
UltralyticsAssistant May 12, 2024
df25307
Merge branch 'master' into remove-manual-accounting
UltralyticsAssistant May 12, 2024
84729a4
Merge branch 'master' into remove-manual-accounting
UltralyticsAssistant May 12, 2024
f13e1af
Merge branch 'master' into remove-manual-accounting
UltralyticsAssistant May 12, 2024
3d46db5
Merge branch 'master' into remove-manual-accounting
UltralyticsAssistant May 13, 2024
e21e701
Merge branch 'master' into remove-manual-accounting
UltralyticsAssistant May 18, 2024
68ff379
Merge branch 'master' into remove-manual-accounting
UltralyticsAssistant May 24, 2024
e0f0110
Merge branch 'master' into remove-manual-accounting
UltralyticsAssistant May 28, 2024
1140453
Merge branch 'master' into remove-manual-accounting
UltralyticsAssistant May 29, 2024
7048f53
Merge branch 'master' into remove-manual-accounting
UltralyticsAssistant May 29, 2024
0faa03c
Merge branch 'master' into remove-manual-accounting
UltralyticsAssistant May 30, 2024
52bf564
Merge branch 'master' into remove-manual-accounting
UltralyticsAssistant Jun 8, 2024
3fe4b9d
Merge branch 'master' into remove-manual-accounting
UltralyticsAssistant Jun 8, 2024
b37e281
Merge branch 'master' into remove-manual-accounting
UltralyticsAssistant Jun 9, 2024
d0c0f77
Merge branch 'master' into remove-manual-accounting
UltralyticsAssistant Jun 16, 2024
59639b3
Merge branch 'master' into remove-manual-accounting
UltralyticsAssistant Jun 16, 2024
a117670
Merge branch 'master' into remove-manual-accounting
UltralyticsAssistant Jun 17, 2024
5faab79
Merge branch 'master' into remove-manual-accounting
UltralyticsAssistant Jun 19, 2024
925d407
Merge branch 'master' into remove-manual-accounting
UltralyticsAssistant Jun 19, 2024
49675db
Merge branch 'master' into remove-manual-accounting
UltralyticsAssistant Jun 20, 2024
1bb4302
Merge branch 'master' into remove-manual-accounting
UltralyticsAssistant Jun 20, 2024
aa9d765
Merge branch 'master' into remove-manual-accounting
UltralyticsAssistant Jun 20, 2024
aea62ca
Merge branch 'master' into remove-manual-accounting
UltralyticsAssistant Jun 20, 2024
8b6d79d
Merge branch 'master' into remove-manual-accounting
UltralyticsAssistant Jun 22, 2024
d99f0be
Merge branch 'master' into remove-manual-accounting
UltralyticsAssistant Jun 30, 2024
2f9a0ba
Merge branch 'master' into remove-manual-accounting
UltralyticsAssistant Jun 30, 2024
93e6aea
Merge branch 'master' into remove-manual-accounting
UltralyticsAssistant Jun 30, 2024
dfe9cbd
Merge branch 'master' into remove-manual-accounting
UltralyticsAssistant Jul 5, 2024
819bf6b
Merge branch 'master' into remove-manual-accounting
UltralyticsAssistant Jul 8, 2024
b91e0fd
Merge branch 'master' into remove-manual-accounting
UltralyticsAssistant Jul 8, 2024
df0f06a
Merge branch 'master' into remove-manual-accounting
UltralyticsAssistant Jul 8, 2024
acc3591
Merge branch 'master' into remove-manual-accounting
UltralyticsAssistant Jul 10, 2024
6cdf59a
Merge branch 'master' into remove-manual-accounting
UltralyticsAssistant Jul 13, 2024
6a93229
Merge branch 'master' into remove-manual-accounting
UltralyticsAssistant Jul 15, 2024
e5b4c52
Merge branch 'master' into remove-manual-accounting
UltralyticsAssistant Jul 15, 2024
a7f25af
Merge branch 'master' into remove-manual-accounting
UltralyticsAssistant Jul 17, 2024
b4eb955
Merge branch 'master' into remove-manual-accounting
UltralyticsAssistant Jul 17, 2024
5c5b82a
Merge branch 'master' into remove-manual-accounting
UltralyticsAssistant Jul 22, 2024
019d527
Merge branch 'master' into remove-manual-accounting
UltralyticsAssistant Jul 23, 2024
30b136a
Merge branch 'master' into remove-manual-accounting
UltralyticsAssistant Jul 25, 2024
6775d7b
Merge branch 'master' into remove-manual-accounting
UltralyticsAssistant Jul 29, 2024
1d4130c
Merge branch 'master' into remove-manual-accounting
UltralyticsAssistant Jul 29, 2024
e3c3c49
Merge branch 'master' into remove-manual-accounting
UltralyticsAssistant Aug 11, 2024
64b6b96
Merge branch 'master' into remove-manual-accounting
UltralyticsAssistant Aug 14, 2024
1bae8d5
Merge branch 'master' into remove-manual-accounting
UltralyticsAssistant Aug 14, 2024
2a826bf
Merge branch 'master' into remove-manual-accounting
UltralyticsAssistant Aug 19, 2024
0b9539b
Merge branch 'master' into remove-manual-accounting
UltralyticsAssistant Aug 20, 2024
73373de
Merge branch 'master' into remove-manual-accounting
UltralyticsAssistant Aug 20, 2024
44728eb
Merge branch 'master' into remove-manual-accounting
UltralyticsAssistant Aug 24, 2024
fa8c344
Auto-format by https://ultralytics.com/actions
UltralyticsAssistant Aug 24, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions export.py
Original file line number Diff line number Diff line change
Expand Up @@ -449,6 +449,7 @@ def transform_fn(data_item):
Quantization transform function.

Extracts and preprocess input data from dataloader item for quantization.

Parameters:
data_item: Tuple with data item produced by DataLoader during iteration
Returns:
Expand Down
9 changes: 7 additions & 2 deletions segment/train.py
Original file line number Diff line number Diff line change
Expand Up @@ -349,8 +349,13 @@ def lf(x):
train_loader.sampler.set_epoch(epoch)
pbar = enumerate(train_loader)
LOGGER.info(
("\n" + "%11s" * 8)
% ("Epoch", "GPU_mem", "box_loss", "seg_loss", "obj_loss", "cls_loss", "Instances", "Size")
"\n"
+ "".join(
[
f"{el:>11}"
for el in ["Epoch", "GPU_mem", "box_loss", "seg_loss", "obj_loss", "cls_loss", "Instances", "Size"]
]
)
)
if RANK in {-1, 0}:
pbar = tqdm(pbar, total=nb, bar_format=TQDM_BAR_FORMAT) # progress bar
Expand Down
31 changes: 19 additions & 12 deletions segment/val.py
Original file line number Diff line number Diff line change
Expand Up @@ -263,18 +263,25 @@ def run(
if isinstance(names, (list, tuple)): # old format
names = dict(enumerate(names))
class_map = coco80_to_coco91_class() if is_coco else list(range(1000))
s = ("%22s" + "%11s" * 10) % (
"Class",
"Images",
"Instances",
"Box(P",
"R",
"mAP50",
"mAP50-95)",
"Mask(P",
"R",
"mAP50",
"mAP50-95)",
s = "".join(
[
f"{el:>11}" if idx > 0 else f"{el:>22}"
for idx, el in enumerate(
[
"Class",
"Images",
"Instances",
"Box(P",
"R",
"mAP50",
"mAP50-95)",
"Mask(P",
"R",
"mAP50",
"mAP50-95)",
]
)
]
)
dt = Profile(device=device), Profile(device=device), Profile(device=device)
metrics = Metrics()
Expand Down
7 changes: 6 additions & 1 deletion train.py
Original file line number Diff line number Diff line change
Expand Up @@ -380,7 +380,12 @@ def lf(x):
if RANK != -1:
train_loader.sampler.set_epoch(epoch)
pbar = enumerate(train_loader)
LOGGER.info(("\n" + "%11s" * 7) % ("Epoch", "GPU_mem", "box_loss", "obj_loss", "cls_loss", "Instances", "Size"))
LOGGER.info(
"\n"
+ "".join(
[f"{el:>11}" for el in ["Epoch", "GPU_mem", "box_loss", "obj_loss", "cls_loss", "Instances", "Size"]]
)
)
if RANK in {-1, 0}:
pbar = tqdm(pbar, total=nb, bar_format=TQDM_BAR_FORMAT) # progress bar
optimizer.zero_grad()
Expand Down
1 change: 0 additions & 1 deletion utils/augmentations.py
Original file line number Diff line number Diff line change
Expand Up @@ -156,7 +156,6 @@ def random_perspective(
):
# torchvision.transforms.RandomAffine(degrees=(-10, 10), translate=(0.1, 0.1), scale=(0.9, 1.1), shear=(-10, 10))
# targets = [cls, xyxy]

"""Applies random perspective transformation to an image, modifying the image and corresponding labels."""
height = im.shape[0] + border[0] * 2 # shape(h,w,c)
width = im.shape[1] + border[1] * 2
Expand Down
1 change: 0 additions & 1 deletion utils/callbacks.py
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,6 @@ def run(self, hook, *args, thread=False, **kwargs):
thread: (boolean) Run callbacks in daemon thread
kwargs: Keyword Arguments to receive from YOLOv5
"""

assert hook in self._callbacks, f"hook '{hook}' not found in callbacks {self._callbacks}"
for logger in self._callbacks[hook]:
if thread:
Expand Down
7 changes: 4 additions & 3 deletions utils/dataloaders.py
Original file line number Diff line number Diff line change
Expand Up @@ -1104,7 +1104,8 @@ def extract_boxes(path=DATASETS_DIR / "coco128"):
def autosplit(path=DATASETS_DIR / "coco128/images", weights=(0.9, 0.1, 0.0), annotated_only=False):
"""Autosplit a dataset into train/val/test splits and save path/autosplit_*.txt files
Usage: from utils.dataloaders import *; autosplit()
Arguments

Arguments:
path: Path to images directory
weights: Train, val, test weights (list, tuple)
annotated_only: Only use images with an annotated txt file
Expand Down Expand Up @@ -1183,7 +1184,7 @@ class HUBDatasetStats:
"""
Class for generating HUB dataset JSON and `-hub` dataset directory.

Arguments
Arguments:
path: Path to data.yaml or data.zip (with data.yaml inside data.zip)
autodownload: Attempt to download dataset if not found locally

Expand Down Expand Up @@ -1314,7 +1315,7 @@ class ClassificationDataset(torchvision.datasets.ImageFolder):
"""
YOLOv5 Classification Dataset.

Arguments
Arguments:
root: Dataset path
transform: torchvision transforms, used by default
album_transform: Albumentations transforms, used if installed
Expand Down
2 changes: 0 additions & 2 deletions utils/general.py
Original file line number Diff line number Diff line change
Expand Up @@ -518,7 +518,6 @@ def check_font(font=FONT, progress=False):

def check_dataset(data, autodownload=True):
"""Validates and/or auto-downloads a dataset, returning its configuration as a dictionary."""

# Download (optional)
extract_dir = ""
if isinstance(data, (str, Path)) and (is_zipfile(data) or is_tarfile(data)):
Expand Down Expand Up @@ -1023,7 +1022,6 @@ def non_max_suppression(
Returns:
list of detections, on (n,6) tensor per image [xyxy, conf, cls]
"""

# Checks
assert 0 <= conf_thres <= 1, f"Invalid Confidence threshold {conf_thres}, valid values are between 0.0 and 1.0"
assert 0 <= iou_thres <= 1, f"Invalid IoU {iou_thres}, valid values are between 0.0 and 1.0"
Expand Down
3 changes: 2 additions & 1 deletion utils/loggers/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -350,7 +350,8 @@ class GenericLogger:
"""
YOLOv5 General purpose logger for non-task specific logging
Usage: from utils.loggers import GenericLogger; logger = GenericLogger(...)
Arguments
Arguments:
opt: Run arguments
console_logger: Console logger
include: loggers to include
Expand Down
14 changes: 7 additions & 7 deletions utils/loggers/clearml/clearml_utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ def __init__(self, opt, hyp):
- Initialize ClearML Task, this object will capture the experiment
- Upload dataset version to ClearML Data if opt.upload_dataset is True
arguments:
Arguments:
opt (namespace) -- Commandline arguments for this run
hyp (dict) -- Hyperparameters for this run
Expand Down Expand Up @@ -133,7 +133,7 @@ def log_scalars(self, metrics, epoch):
"""
Log scalars/metrics to ClearML.
arguments:
Arguments:
metrics (dict) Metrics in dict format: {"metrics/mAP": 0.8, ...}
epoch (int) iteration number for the current set of metrics
"""
Expand All @@ -145,7 +145,7 @@ def log_model(self, model_path, model_name, epoch=0):
"""
Log model weights to ClearML.
arguments:
Arguments:
model_path (PosixPath or str) Path to the model weights
model_name (str) Name of the model visible in ClearML
epoch (int) Iteration / epoch of the model weights
Expand All @@ -158,7 +158,7 @@ def log_summary(self, metrics):
"""
Log final metrics to a summary table.
arguments:
Arguments:
metrics (dict) Metrics in dict format: {"metrics/mAP": 0.8, ...}
"""
for k, v in metrics.items():
Expand All @@ -168,7 +168,7 @@ def log_plot(self, title, plot_path):
"""
Log image as plot in the plot section of ClearML.
arguments:
Arguments:
title (str) Title of the plot
plot_path (PosixPath or str) Path to the saved image file
"""
Expand All @@ -183,7 +183,7 @@ def log_debug_samples(self, files, title="Debug Samples"):
"""
Log files (images) as debug samples in the ClearML task.
arguments:
Arguments:
files (List(PosixPath)) a list of file paths in PosixPath format
title (str) A title that groups together images with the same values
"""
Expand All @@ -199,7 +199,7 @@ def log_image_with_boxes(self, image_path, boxes, class_names, image, conf_thres
"""
Draw the bounding boxes on a single image and report the result as a ClearML debug sample.
arguments:
Arguments:
image_path (PosixPath) the path the original image file
boxes (list): list of scaled predictions in the format - [xmin, ymin, xmax, ymax, confidence, class]
class_names (dict): dict containing mapping of class int to class name
Expand Down
12 changes: 6 additions & 6 deletions utils/loggers/wandb/wandb_utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ def __init__(self, opt, run_id=None, job_type="Training"):
- Upload dataset if opt.upload_dataset is True
- Setup training processes if job_type is 'Training'
arguments:
Arguments:
opt (namespace) -- Commandline arguments for this run
run_id (str) -- Run ID of W&B run to be resumed
job_type (str) -- To set the job_type for this run
Expand Down Expand Up @@ -90,7 +90,7 @@ def setup_training(self, opt):
- Update data_dict, to contain info of previous run if resumed and the paths of dataset artifact if downloaded
- Setup log_dict, initialize bbox_interval
arguments:
Arguments:
opt (namespace) -- commandline arguments for this run
"""
Expand Down Expand Up @@ -120,7 +120,7 @@ def log_model(self, path, opt, epoch, fitness_score, best_model=False):
"""
Log the model checkpoint as W&B artifact.
arguments:
Arguments:
path (Path) -- Path of directory containing the checkpoints
opt (namespace) -- Command line arguments for this run
epoch (int) -- Current epoch number
Expand Down Expand Up @@ -159,7 +159,7 @@ def log(self, log_dict):
"""
Save the metrics to the logging dictionary.
arguments:
Arguments:
log_dict (Dict) -- metrics/media to be logged in current step
"""
if self.wandb_run:
Expand All @@ -170,7 +170,7 @@ def end_epoch(self):
"""
Commit the log_dict, model artifacts and Tables to W&B and flush the log_dict.
arguments:
Arguments:
best_result (boolean): Boolean representing if the result of this evaluation is best or not
"""
if self.wandb_run:
Expand All @@ -197,7 +197,7 @@ def finish_run(self):

@contextmanager
def all_logging_disabled(highest_level=logging.CRITICAL):
"""source - https://gist.github.com/simon-weber/7853144
"""Source - https://gist.github.com/simon-weber/7853144
A context manager that will prevent any logging messages triggered during the body from being processed.
:param highest_level: the maximum logging level in use.
This would only need to be changed if a custom level greater than CRITICAL is defined.
Expand Down
8 changes: 3 additions & 5 deletions utils/metrics.py
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,6 @@ def ap_per_class(tp, conf, pred_cls, target_cls, plot=False, save_dir=".", names
# Returns
The average precision as computed in py-faster-rcnn.
"""

# Sort by objectness
i = np.argsort(-conf)
tp, conf, pred_cls = tp[i], conf[i], pred_cls[i]
Expand Down Expand Up @@ -103,7 +102,6 @@ def compute_ap(recall, precision):
# Returns
Average precision, precision curve, recall curve
"""

# Append sentinel values to beginning and end
mrec = np.concatenate(([0.0], recall, [1.0]))
mpre = np.concatenate(([1.0], precision, [0.0]))
Expand Down Expand Up @@ -137,6 +135,7 @@ def process_batch(self, detections, labels):
Return intersection-over-union (Jaccard index) of boxes.
Both sets of boxes are expected to be in (x1, y1, x2, y2) format.
Arguments:
detections (Array[N, 6]), x1, y1, x2, y2, conf, class
labels (Array[M, 5]), class, x1, y1, x2, y2
Expand Down Expand Up @@ -233,7 +232,6 @@ def bbox_iou(box1, box2, xywh=True, GIoU=False, DIoU=False, CIoU=False, eps=1e-7
Input shapes are box1(1,4) to box2(n,4).
"""

# Get the coordinates of bounding boxes
if xywh: # transform from xywh to xyxy
(x1, y1, w1, h1), (x2, y2, w2, h2) = box1.chunk(4, -1), box2.chunk(4, -1)
Expand Down Expand Up @@ -279,14 +277,15 @@ def box_iou(box1, box2, eps=1e-7):
Return intersection-over-union (Jaccard index) of boxes.
Both sets of boxes are expected to be in (x1, y1, x2, y2) format.
Arguments:
box1 (Tensor[N, 4])
box2 (Tensor[M, 4])
Returns:
iou (Tensor[N, M]): the NxM matrix containing the pairwise
IoU values for every element in boxes1 and boxes2
"""

# inter(N,M) = (rb(N,M,2) - lt(N,M,2)).clamp(0).prod(2)
(a1, a2), (b1, b2) = box1.unsqueeze(1).chunk(2, 2), box2.unsqueeze(0).chunk(2, 2)
inter = (torch.min(a2, b2) - torch.max(a1, b1)).clamp(0).prod(2)
Expand All @@ -304,7 +303,6 @@ def bbox_ioa(box1, box2, eps=1e-7):
box2: np.array of shape(nx4)
returns: np.array of shape(n)
"""

# Get the coordinates of bounding boxes
b1_x1, b1_y1, b1_x2, b1_y2 = box1
b2_x1, b2_y1, b2_x2, b2_y2 = box2.T
Expand Down
1 change: 0 additions & 1 deletion utils/segment/augmentations.py
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,6 @@ def random_perspective(
):
# torchvision.transforms.RandomAffine(degrees=(-10, 10), translate=(.1, .1), scale=(.9, 1.1), shear=(-10, 10))
# targets = [cls, xyxy]

"""Applies random perspective, rotation, scale, shear, and translation augmentations to an image and targets."""
height = im.shape[0] + border[0] * 2 # shape(h,w,c)
width = im.shape[1] + border[1] * 2
Expand Down
3 changes: 0 additions & 3 deletions utils/segment/general.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,6 @@ def crop_mask(masks, boxes):
- masks should be a size [n, h, w] tensor of masks
- boxes should be a size [n, 4] tensor of bbox coords in relative point form
"""

n, h, w = masks.shape
x1, y1, x2, y2 = torch.chunk(boxes[:, :, None], 4, 1) # x1 shape(1,1,n)
r = torch.arange(w, device=masks.device, dtype=x1.dtype)[None, None, :] # rows shape(1,w,1)
Expand All @@ -33,7 +32,6 @@ def process_mask_upsample(protos, masks_in, bboxes, shape):
return: h, w, n
"""

c, mh, mw = protos.shape # CHW
masks = (masks_in @ protos.float().view(c, -1)).sigmoid().view(-1, mh, mw)
masks = F.interpolate(masks[None], shape, mode="bilinear", align_corners=False)[0] # CHW
Expand All @@ -51,7 +49,6 @@ def process_mask(protos, masks_in, bboxes, shape, upsample=False):
return: h, w, n
"""

c, mh, mw = protos.shape # CHW
ih, iw = shape
masks = (masks_in @ protos.float().view(c, -1)).sigmoid().view(-1, mh, mw) # CHW
Expand Down
3 changes: 1 addition & 2 deletions utils/triton.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,10 +17,9 @@ class TritonRemoteModel:

def __init__(self, url: str):
"""
Keyword arguments:
Keyword Arguments:
url: Fully qualified address of the Triton server - for e.g. grpc://localhost:8000
"""

parsed_url = urlparse(url)
if parsed_url.scheme == "grpc":
from tritonclient.grpc import InferenceServerClient, InferInput
Expand Down
7 changes: 6 additions & 1 deletion val.py
Original file line number Diff line number Diff line change
Expand Up @@ -319,7 +319,12 @@ def run(
if isinstance(names, (list, tuple)): # old format
names = dict(enumerate(names))
class_map = coco80_to_coco91_class() if is_coco else list(range(1000))
s = ("%22s" + "%11s" * 6) % ("Class", "Images", "Instances", "P", "R", "mAP50", "mAP50-95")
s = "".join(
[
f"{el:>11}" if idx > 0 else f"{el:>22}"
for idx, el in enumerate(["Class", "Images", "Instances", "P", "R", "mAP50", "mAP50-95"])
]
)
tp, fp, p, r, f1, mp, mr, map50, ap50, map = 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0
dt = Profile(device=device), Profile(device=device), Profile(device=device) # profiling times
loss = torch.zeros(3, device=device)
Expand Down