How to add horizontal lines showing facilities for all groups in ggplot2?

Question

Is there a way to place horizontal lines using group tools on a chart without first creating summary data? I know this works, but I believe there should be a way to do this with just one ggplot2.

library(dplyr)
library(ggplot2)
X <- data_frame(
  x = rep(1:5, 3),
  y = c(rnorm(5, 5, 0.5),
        rnorm(5, 3, 0.3),
        rnorm(5, 4, 0.7)),
  grp = rep(LETTERS[1:3], each = 5))

X.mean <- X %>%
  group_by(grp) %>%
  summarize(y = mean(y))

X %>%
  ggplot(aes(x = x, y = y, color = grp)) +
  geom_point(shape = 19) +
  geom_hline(data = X.mean, aes(group = grp, yintercept = y, color = grp)) +
  background_grid()

Answer 1

Extension of my comment:

ggplot(X, aes(x = x, y = y, color = grp)) +
  geom_point(shape = 19) +
  stat_smooth(method="lm", formula=y~1, se=FALSE)+
  theme_bw()

Thus, a linear model with a constant term that returns the average value is applied. Sign this answer for the main idea.

Edit: response to a very clever OP proposal.

It looks like you can use quantile regression to create medians!

library(quantreg)
ggplot(X, aes(x = x, y = y, color = grp)) +
  geom_point(shape = 19) +
  stat_smooth(method="rq", formula=y~1, se=FALSE)+
  theme_bw()

stat_smooth(method=..., ...) , , predict(...). , rq(...) rq predict.rq(...). se=TRUE, .